Planet Classpath
I recently upgraded my RSS reader from the older version I was still using to the current version. That turned out to be a mistake. The new version was even more broken than the old version, so I decided it was time to switch. I remembered RSSOwl from several years ago when I tested it on IKVM (it uses the Eclipse Standard Widget Toolkit, so like Eclipse it was a good test app back when AWT support was completely useless).
I downloaded the most recent version and played with it and it appeared to suit my needs. Of course, after I decided that I was going to start using it, I wanted to run it on IKVM and not in dynamic mode, but compiled with ikvmc. Fortunately, RSSOwl uses OSGi in much the same way as Eclipse, so I was able to reuse the work I did to get Eclipse to compile with ikvmc.
To play along at home, follow these instructions (on Windows):
unzip rssowl-2.0.5.win32.zipcd rssowlunzip ..\ikvmbin-0.44.0.5.zipunzip ..\rssowl-clr.zipmkWhile the 0.44 release candidates have been baking, I've been working on 0.45. There are some interesting changes related to resource handling and stub classes.
Resources
Previously, if you looked at the URL returned by ClassLoader.getResource() for an ikvmc compiled assembly you see something ugly like this:
ikvmres://IKVM.OpenJDK.Jdbc,%20Version=0.44.0.5,%20Culture=neutral,%20PublicKeyToken=13235d27fcbfff58/META-INF/services/java.sql.Driver
Now with 0.45, you see:
jar:file:/C:/.virtual-ikvm-home/assembly/IKVM.OpenJDK.Jdbc__0.45.3887.0__13235d27fcbfff58/resources/resources.jar!/META-INF/services/java.sql.Driver
This is also a bit strange, because C:\.virtual-ikvm-home doesn't actually exist, but it is the IKVM Virtual File System directory that was introduced with the switch to OpenJDK, to facilitate the fact that OpenJDK likes to load lots of files from the installation directory.
Starting with 0.45, the virtual file system is also used to load resources and stub classes. When you compile your jar with ikvmc, the resources in the jar will be copied into a new jar (usually with the same name) and that jar will be attached as a managed resource to the target assembly. This resource is projected into VFS and the normal Java resource loading mechanism is (more or less) used to load resources from the jar.
This has two main advantages. The first is that this makes it more likely that Java code that makes various assumptions about being being able to explicitly open a resource jar, will work. The second is that this method of storing resources, usually results in smaller assemblies.
Another benefit of this change is that I finally fixed the issue with ikvmc skipping resources due to name clashes. Previously there was only a single resource namespace per assembly, but now an assembly can contain multiple resource jars.
Stub Classes
Some Java code requires .class files for system classes. This is usually because they want to do dynamic code generation and Java's reflection isn't really good enough for that. For a long time IKVM has supported this by dynamically creating the .class files (in a runtime equivalent of ikvmstub) whenever code tried to load a resource that ended with .class and a corresponding type was found. This used the same ikvmres protocol mechanism as normal resources. With this snapshot, the stub classes have also moved to VFS. They are still generated on demand, but they are now accessible via the Java file IO APIs. This means that the sun.boot.class.path property can now point to VFS and that Java code, like javac, that depends on sun.boot.class.path will now work.
You can now build a working javac.exe like this:
ikvmc -main:com.sun.tools.javac.Main -out:javac.exe -r:IKVM.OpenJDK.Tools.dll
The resulting javac.exe will be very small (4KB), because all the code is in IKVM.OpenJDK.Tools.dll (the equivalent of tools.jar).
Changes:
Binaries available here: ikvmbin-0.45.3887.zip
I have a confession to make: it seems that there will be no CACAO 1.0 release, and I’m to blame for that. Maintaining two branches of CACAO has become tiresome and quite boring over time, and I never fully understood what the 1.0 branch was there for, anyway. So when yet another problem with IcedTea (which was using CACAO 0.99.4 at the time) popped up which could trivially be solved by switching to the “default” branch, I suggested that IcedTea would be better off just using the Mercurial version, and this is what they did. Now, as this version identifies itself as 1.1, I don’t see the point of releasing a 1.0 version from a branch that nobody cares about (or should care about, IMHO) anymore.
As of build 105 of JDK 7, the compiler and libraries have support for Project Coin's try-with-resources statement (ARM blocks).
We're curious to get feedback on how try-with-resources works in practice, so please download the build, use the feature to on your favorite Closeable type, and report back on how it goes.
Another fortune, I promise two real cool blog entries in the next couple of days :) (with screenshots and all the fun things!)
... Where was Stac Electronics when Microsoft invented Doublespace? Where were Xerox and Apple when Microsoft invented the GUI? Where was Apple's QuickTime when Microsoft invented Video for Windows? Where was Spyglass Inc.'s Mosaic when Microsoft invented Internet Explorer? Where was Sun when Microsoft invented Java?
I wrote a while ago about my blog-reading woes. Those woes are now over!
Lars Magne Ingebrigtsen, of Gnus and gmane fame, has now brought us gwene — an RSS to NNTP gateway. You enter the feeds you want to read, and soon they show up as newsgroups in gmane. Thanks also should go to Ted Zlanatov, for bringing this up on the gmane discussion list, and thus getting it all rolling.
I haven’t quite retired my rss2email cron job, but that is mostly out of laziness. Any day now.
Normally I am unhappy about the whole SaaS trend, but gmane gets a pass. I am not sure if it is because Lars seems trustworthy, or because NNTP is so obviously a fringe interest, or because gmane is at least theoretically replaceable in the event of the worst.
Seems Oracle bought Sun to become a java patent troll. Trying to destroy the alternative free java implementation that is part of android. Sun used to be agnostic towards Free Software in the past, then became a huge fan on java liberation day. Now that Oracle is in control and starts its quest to destroy the free java world, we are back to the dark ages. So, now what?
Oracle is still distributing a free version of java itself through OpenJDK on which IcedTea is based. Implementations derived from that source base seem safe against copyright and patent claims as long as one follows the obligations of the GPL. That is of course only for patent and copyrights Oracle holds or can pass on (through its stewardship of the JCP). To protect against claims by unrelated companies or when you have a free implementation of java not based on code distributed by Oracle, like GNU Classpath, try to get your implementation covered by a Free Software friendly patent pool. For example gcj/libgcj/GNU Classpath (as are some parts of the apache and eclipse java stacks) are part of the “System Components” of OIN (and Oracle has joined OIN).
Finally if you contribute to any Sun/Oracle java implementation demand that they change their Contributor Agreement to be truly reciprocal, not just for copyrights, but also for any patent claims covering the project you contribute to. So that anybody that wants to share the project you contributed to will always and irrevocably get all the rights to do that (not just for the GPL version). Point 3 in the current Sun/Oracle Contributor Agreement isn’t reciprocal, you grant a perpetual, irrevocable, non-exclusive, worldwide, no-charge, royalty-free license to any patent claims you might have to Sun/Oracle, but they are not granting back to you or the wider community any they hold on the project as a whole.
Humorix Holiday Gift Idea #5 AbsoluteZero(tm) Cryogenic Refrigerator $29,999.95 for economy model at Cryo-Me-A-River, Inc. The pundits have been hyping new technology allowing your home appliances to have Internet access. Most people aren't too keen with the thought of their refrigerator sharing an IP address with their can opener. But with the new AbsoluteZero(tm) Refrigerator, that might change. This is not a fridge for your food; it's a fridge for your overclocked, overheating CPU. You stick your computer inside, bolt the door shut, turn the temperature down to 5 degrees Kelvin, and you've got the perfect environment for accelerating your CPU to 1 Terahertz or more. This cryogenic cooling system may not actually reach absolute zero, but it comes mighty close. Unfortunately, the AbsoluteZero(tm) is the size of a small house, consumes a constant stream of liquid nitrogen, and requires it's own nuclear reactor (not included). But that's a small price to pay for the ability to play Quake 3 at 100,000 frames per second.
WordPress 3.0 integrated Multisite support! I only maintain two blogs, but it was already handy to put them under the same “Super Admin” install. WordPress really is pretty smooth these days and highly customizable.
As previously considered, the specification of the try-with-resources statement has been updated to require uniform suppression of all throwables. That is, the semantics of the code around the implicitly generated calls to close was previously
try {
#resource.close();
} catch(Exception #suppressedException) {
#primaryException.addSuppressedException(#suppressedException);
}
and has been replaced with
try {
#resource.close();
} catch(Throwable #suppressedException) {
#primaryException.addSuppressedException(#suppressedException);
}
(The initial implementation actually implemented the latter semantics; a test has been added to verify the behavior.)
I had the opportunity to work on a nasty little defect from home this afternoon. Some background:
Fast foward to yesterday where I've successfully been able to cross-compile and osgi-ify libbatthew , java-dbus , and bluecove. With all that stuff built, I wanted to run a quick test and make sure all the build mojo was worked out (linking, cross compilation, bluez4 compatibility, openjdk-6 & classpath compaibility). I planned to use the bluecove-tester-app.jar to test and make sure that java/jni/bluez interaction works at the basic device-scan level. I stick all the jni libs in the right places and give it a whirl aaaand:
root@bug20:~# java -Djava.library.path=/usr/lib/jni/ -cp /usr/share/java/com.buglabs.bug.jni.bluecove.jar:/home/root/junit.jar:. -jar bluecove-tester-app.jar --console
BlueCove tester Console application (keyboard codes the same as in MIDP application) 2 - Start Client 3 - Stop Client 5 - Start Server 6 - Stop Server * - Run Discovery 7 - Services Search d - toggle BlueCove Debug T - Start TCK Agent q - Quit * error start error java.lang.UnsatisfiedLinkError: com.intel.bluetooth.BluetoothStackBlueZ.nativeGetDeviceID(IJ)I java.lang.UnsatisfiedLinkError: com.intel.bluetooth.BluetoothStackBlueZ.nativeGetDeviceID(IJ)I at com.intel.bluetooth.BluetoothStackBlueZ.nativeGetDeviceID(Native Method) at com.intel.bluetooth.BluetoothStackBlueZ.initialize(BluetoothStackBlueZ.java:136) at com.intel.bluetooth.BlueCoveImpl.setBluetoothStack(BlueCoveImpl.java:949) at com.intel.bluetooth.BlueCoveImpl.detectStack(BlueCoveImpl.java:482) at com.intel.bluetooth.BlueCoveImpl.access$500(BlueCoveImpl.java:65) at com.intel.bluetooth.BlueCoveImpl$1.run(BlueCoveImpl.java:1020) at java.security.AccessController.doPrivileged(Native Method) at com.intel.bluetooth.BlueCoveImpl.detectStackPrivileged(BlueCoveImpl.java:1018) at com.intel.bluetooth.BlueCoveImpl.getBluetoothStack(BlueCoveImpl.java:1011) at javax.bluetooth.LocalDevice.getLocalDeviceInstance(LocalDevice.java:75) at javax.bluetooth.LocalDevice.getLocalDevice(LocalDevice.java:95) at net.sf.bluecove.TestResponderCommon.initLocalDeviceConfig(TestResponderCommon.java:71) at net.sf.bluecove.TestResponderCommon.startLocalDevice(TestResponderCommon.java:66) at net.sf.bluecove.TestResponderClient.(TestResponderClient.java:124) at net.sf.bluecove.Switcher.createClient(Switcher.java:269) at net.sf.bluecove.Switcher.createClient(Switcher.java:262) at net.sf.bluecove.Switcher.startDiscovery(Switcher.java:326) at net.sf.bluecove.se.Console.main(Console.java:70) at net.sf.bluecove.se.Main.runConsole(Main.java:86) at net.sf.bluecove.se.Main.main(Main.java:54)
...
Ugly. Looks exactly like my linker problem before. Thankfully with the angstrom feeds being part of ipkg's config by default, it was easy enough to install ldd and determine if it linked correctly. It didn't:
root@bug20:~# ldd libbluecove_arm.so libc.so.6 => /lib/libc.so.6 (0x4000f000) /lib/ld-linux.so.3 (0x2a000000) root@bug20:~#
Okay so recompile it and make sure it links with the appropriate -L flags to the cross compiler and now:
root@bug20:~# ldd libbluecove_arm.so libbluetooth.so.3 => /usr/lib/libbluetooth.so.3 (0x4000f000) libc.so.6 => /lib/libc.so.6 (0x4002a000) /lib/ld-linux.so.3 (0x2a000000) root@bug20:~#
Looks good! Except:
root@bug20:~# java -Djava.library.path=/usr/lib/jni/ -cp /usr/share/java/com.buglabs.bug.jni.bluecove.jar:/home/root/junit.jar:. -jar bluecove-tester-app.jar --console
BlueCove tester Console application (keyboard codes the same as in MIDP application) 2 - Start Client 3 - Stop Client 5 - Start Server 6 - Stop Server * - Run Discovery 7 - Services Search d - toggle BlueCove Debug T - Start TCK Agent q - Quit * error start error java.lang.UnsatisfiedLinkError: com.intel.bluetooth.BluetoothStackBlueZ.nativeGetDeviceID(IJ)I
WTF!!! So I recheck everything and even recheck my knowledge of linking and gcc flags and am I even sure my parents are my parents? You know, one of those kinds of afternoons. I resigned to let it go and resume fixing it today, remotely. Remotely remotely. I mean using openjdk-6 built-in sweet sweet remote debugging capabilities. The fact that this is BUILT IN to openjdk-6 (at least for the zero implementation) means you just follow the simple java remote debugging instructions for eclipse and that's it. I can even do it via our VPN and access the BUG sitting on my desk in the office to test. 30 seconds of setup and you just start stepping through code. It's beautiful.
One thing that perturbed me was that the exception stacktrace gave an UnsatisfiedLinkError and the last method in the stacktrace was BluetoothStackBlueZ.nativeGetDeviceID(IJ)I . But as I step through the code, I see:
private int nativeGetDeviceID(int id, long findLocalDeviceBTAddress) throws BluetoothStateException {
No native declaration? That means... it's uh, not native. But wtf is going on here? Who says it's native?
root@bug20:~# zipinfo bluecove-tester-app.jar | grep BlueZ.class -rw-r--r-- 2.3 unx 20622 bx defN 25-Dec-08 17:55 com/intel/bluetooth/BluetoothStackBlueZ.class
Why you little sneaky jar you. I see what's going on here. In some old implementation of bluecove, this nativeGetDeviceID was uh, NATIVE. In my newer (2.1.0) it's NOT native. For backwards compatibility they kept the method name "nativeGetDeviceID" despite the fact that it's not a jni call anymore.
But how can I be certain? I mean, I demand satisfaction. I need to know that this is indeed the cause of my problems. Naturally, the jar is binary-only, meaning I can't just inspect the source for the godforsaken line. But I need to know! So I google around for a decompiler in linux and behold: JD-GUI for linux! Sure enough:
ooh I so have you now. You little defect you:
private native int nativeGetDeviceID(int paramInt, long paramLong) throws BluetoothStateException;
Next time I'll uh, just download the tester jar without the mind-bendingly-subtle divergent API changes pre-bundled. That could have saved me a lot of pain, but then it wouldn't have been so rewarding to figure out, now would it?
Some days ago I stumbled across a Java statement which I thought was trivial at first, only to discover that I had no idea. I have a reasonable understanding of what a JVM does and how Java bytecode is executed. But as this example shows once again, that doesn’t necessarily spread to the Java programming language. The snippet below should explain my point.
int lorem = 1, ipsum = 2, dolor = 3;
if (lorem == (lorem = ipsum))
f();
if ((ipsum = dolor) == ipsum)
g();
Which of the above two methods f() and g() is actually invoked? Can you tell without compiling the code? Possible answers are:
f() is invoked and the call-site of g() is dead code.g() is invoked and the call-site of f() is dead code.Ironically, I finally understood what was going on after looking at the generated bytecode (good old javap is your friend). I am not posting the disassembled code because that would spoil the fun. But once you look at it, the answer appears to be quite obvious.
We are pleased to announce a new major release of IcedTea7, 1.13.
This update contains the following security updates:
The IcedTea project provides a harness to build the source code from OpenJDK6 using Free Software build tools. It also includes the only Free Java plugin and Web Start implementation, and support for additional architectures over and above x86, x86_64 and SPARC via the Zero assembler port.
—————–
The tarball can be downloaded from:
The following people helped with this release:
Gary Benson, Deepak Bhole, Andrew John Hughes, Nobuhiro Iwamatsu, Matthias Klose, Omair Majid, Edward Nevill, Xerxes Rånby, Stefan Ring, Pavel Tisnovsky, Jon VanAlten, Mark Wielaard, Man Lung Wong
We would also like to thank the bug reporters and testers!
To get started:
$ tar xzf icedtea-1.13.tar.gz $ cd icedtea-1.13
Full build requirements and instructions are in INSTALL:
$ ./configure [--enable-zero --with-openjdk --enable-pulse-java --enable-systemtap ...] $ make
[Reposted on behalf of Matthias Klose, release manager for 1.8]
We are proud to announce the release of IcedTea6 1.8.1.
This update contains the following security updates:
The IcedTea project provides a harness to build the source code from OpenJDK6 using Free Software build tools. It also includes the only Free Java plugin and Web Start implementation, and support for additional architectures over and above x86, x86_64 and SPARC via the Zero assembler port.
The tarball can be downloaded here:
The following people helped with this release: Gary Benson, Deepak Bhole, Andrew John Hughes, Mark Wielaard, Matthias Klose, Omair Majid, Pavel Tisnovsky, Xerxes Rånby, Jon VanAlten, Man Lung Wong, and many others.
We would also like to thank the bug reporters and testers!
To get started:
$ hg clone http://icedtea.classpath.org/hg/release/icedtea6-1.8.1 $ cd icedtea6-1.8.1
Full build requirements and instructions are in INSTALL:
$ ./configure [--enable-visualvm --with-openjdk --enable-pulse-java --enable-systemtap --enable-nss ...] $ make
Hello guys, well when I applied to Google Summer of Code I had no idea on how it was going to change my life. So far not only I have been learning new things, but also meeting awesome developers. The climax of this was when we (brazillian students) asked for a small financial help to Google to take us to FISL (Forum Internacional de Software Livre – International Free Software Forum) in Brazil. Google did the amazing favor to help us so that we can learn/spread free software to others. We are in huge debt with them, and we’re amazed by their concerns about free software ideals. We managed not only to meet there (#gsoc-br) but also we managed to meet excellent FOSS developers there and had a lecture about the GsoC experience in FISL.
There I met the remaining gsocers, we shared our difficulties and we exchanged tips on how to solve problems. They all joked on me, :/, when I said that I preferred svn over dscm, like git or mercurial. But at the same time they gave me a very brief and informal talk/introduction to git (which I’m kinda liking 
). I talked also a lot about X and XCB with friend and hackers there since it’s directly related to my proposal.
GSoCers
I had a chance to meet some badass developers (like ‘Maddog’) and get amazing talks (like Glassfigh in OSGi bundles and What’s new on OpenJDK 7). I took some photos and the most cool was with the GNU.
Overall, It was a amazing experience, and I want to thank again Google, specially Carol (which is so kind =]), all the cool guys there (FISL) and my mentor Mario Torre (which understand that I’m a little behind but getting up the schedule now =).
GNU and Me
Does JamVM still produce those quick and fast startup times when using the OpenJDK class libraries compared to the fast startup times obtainable when using GNU Classpath classes?[best of 3 runs]
rob@traken:~/JAM/tests$ time /usr/lib/jvm/jamvm-openjdk/jre/bin/java -showversion hello
java version "1.6.0_0"
OpenJDK Runtime Environment (IcedTea6 1.6.1) (6b16-1.6.1-3ubuntu1)
JamVM (build 1.5.5-openjdk, inline-threaded interpreter)
Hello World!
real 0m0.046s
user 0m0.030s
sys 0m0.000s
rob@traken:~/JAM/tests$ time jamvm -showversion hello
java version "1.5.0"
JamVM version 1.5.5-devel
Copyright (C) 2003-2010 Robert Lougher
...
Build information:
Execution Engine: inline-threaded interpreter
Compiled with: gcc 4.5.0 20100211 (experimental)
Boot Library Path: /usr/local/classpath/lib/classpath
Boot Class Path: /usr/local/jamvm/share/jamvm/classes.zip:/usr/local/classpath/share/classpath/glibj.zip
Hello World!
real 0m0.048s
user 0m0.030s
sys 0m0.020s
Will JamVM still support GNU Classpath, as CACAO does?Yes, most definitely. I still consider GNU Classpath as JamVM's main class-library as it's where my chief loyalty lies. On more practical grounds, even after the OpenJDK port is functionally complete it will still be a long time before it is as tested as JamVM/GNU Classpath. FWIW, many embedded systems seem to be quite happy with GNU Classpath. GNU Classpath is considerably smaller "out of the box" and much easier to build...
The upcoming GSPdf version can be used seamlessly to do Print Preview in GNUstep. The provision to do that has long been in GNUstep's printing code, but no application exploited it.
GSPdf is undergoing quite some restructuring and is now a true document-based application. This allows it to register properly for the post script viewing task. Activationg GhostScript's anti-aliasing is supported too (look at the screenshot, where Graphos itself is running without).
GSPdf is an interface to ghostscript as gv is, so the displayed quality should remain the same.
Hello guys, sorry for the lack of updates, been a busy week here.
The progress so far is that I’m ending up the refactoring in Escher (already finished the core java pkgs) which lend me changing almost all classes in XPeer @ GNU Classpath, I’m waiting right now to submit this patch and some review of my mentor (Mario Torre) so I can make it a best refactoring.
So far:
)One of the things that I changed in XPeer too was the lazy connection. IMHO there is no need to wait for someone to getDisplay() to make all the overhead of connecting to the Xserver. Since usually after the instantiation of the Xtoolkit class there will be use it I did the start right ahead. This change made *somehow* the start of the gui way faster. Am I missing the reason behind lazy connection?
On Monday July 5th starting at 6 PM there will be a Fedora 13 release party at Seneca College’s York University campus. More details are available here:
http://blog.chris.tylers.info/index.php?/archives/238-Fedora-13-Release-Event.html
and
https://fedoraproject.org/wiki/Release_Party_F13_Toronto
All are welcome to attend. I look forward to seeing new and existing Fedora users alike. Bring your laptops to try Fedora 13 if you haven’t already. We’ll also have installation media available and a machine set up for creating live Fedora 13 USB sticks.
On Tuesday, Red Hat hosted an Eclipse DemoCamp at our Toronto office to celebrate the Helios release.
Over 30 people crammed into our conference room to hear from these intrepid presenters:
It was great to have diversity in both topics of presentation and attendees. Two attendees came from a long way away just to attend the DemoCamp: Ian “Ctrl-awesome” Bull from Victoria, BC, Canada; and Alex Kurtakov from Pazardzhik, Bulgaria! Unlike the Guelph DemoCamp when I asked if there were any C or C++ programmers in the room and got one “sort of” from the back, we had some actual C++ people in attendance! There was also representation from academia including Seneca College and the University of Toronto.
Thankfully it wasn’t too hot this year and no one melted (at least as far as I know). After the presentations, we enjoyed some refreshments compliments of the Eclipse Foundation; thanks, Lynn & Ralph!
I’d like to extend a big thank you to the presenters for showing off their wares and to the attendees for spending their evening with us. Ralph mentioned the European Eclipse Stammtisch gatherings … I wonder if we could get something like that going in Toronto?
All the photos taken by me available here: HeliosDemoCamp
GWorkspace 0.8.8 is out! Many thanks to all those who helped with testing or contributed with patches.
GWorkspace is GNUstep's standard workspace manager.
Among the improvements:
The compulsory screenshot shows GWorkspace running with the Neos theme from GAP.
Just some weeks ago this so called security expert (and Google employee) disclosed security problems in Java Webstart,today he disclosed security problem in Windows Help. The problem is not so much that he discloses security issues, but the way he does it. The pattern seems to be similar in both cases. He notifies the company of the security issue, giving them some time (in Java’s case it was at least a month) and then goes on to publish the full disclosure just a couple of days later for idiotic reasons. This misleads the security teams of the companies in the first place and gives them no realistic chance to address the issue in a timely and professional manner, leaving millions of users vulnerable. Of course since everybody loves Google and hates Microsoft and Oracle, he gets lots of applause for this, which is most likely exactly what he wants. If I were Google I would fire him instead for repeated irresponsible and unprofessional handling of security issues. Rant over.
An IcedTea build of OpenJDK using Shark passed the Java SE 6 TCK today. Fedora 12, x86_64, LLVM 2.6, icedtea6-7674917fa451. Dr Fun is here!
Today marks my first anniversary at Sun/Oracle. Last year at this time I arrived in Santa Clara, then came JavaOne, then came 2 weeks with my new team at the east coast and then came a really cool year working in the deployment team (great people). Lots of things happened, in October Selma (my 3rd child) was born, in January the Sun/Oracle deal was finally approved, some weeks ago we had a pretty bad security issue, etc etc. In Germany, Sun is still Sun (officially) … and I won’t be experiencing Oracle anymore (although I am already experiencing Oracle indirectly through my US collegues). Today I resigned my position at Sun and starting in July I will join David to work at JP Morgan in Geneva. Sun was a great time and I wish it had lasted longer, but life has different plans with me 
The spec for jtreg calls for tests to execute in a scratch directory which can be cleared before every test. Early versions of Java™ did not provide the ability to change the current directory when executing a command in a sub process, and so the first versions of jtreg were split into a complicated shell script and a Java program using the JavaTest™ test framework. The shell script analyzed the command line arguments, determined the scratch directory to be used, and set it as the current directory before finally executing the test harness itself.
Even back then, there was support for two modes: "othervm", being the ability to run every test in its own JVM, different from (other than) the JVM used to run the test harness, and "samevm", being the ability to run every test in the same JVM used to run the test harness. However, to begin with, the use of "othervm" was strongly recommended, because the risk of tests affecting one another was quite high, and the abilility to recover from over-zealous tests was quite limited. And, although "samevm" was conceptually well defined for running API tests (@run main Foo, etc) it was less well defined for compilation tests (@compile, etc) and there is still some legacy code in javac that was added to work around the limitations (e.g. -XDstdout, to force the compiler to write to the stream used by the jtreg @compile/ref= option.)
As jtreg evolved, the maintenance of the wrapper script became increasingly difficult. Eventually, the script was translated into Java code, and merged with the harness itself. However, the issue of the current directory remained an issue, and so after analyzing the command line arguments, jtreg performed a check to see if the JVM being used was acceptable for the execution of the test run itself. This is primarily an issue for samevm mode, when the correct current directory must be set, the correct version of Java must be in use, and so on. If any of the checks failed, jtreg restarted itself using the correct parameters -- the correct version of Java, the correct current directory and so on. In this case, the Java code was behaving as "a better wrapper script", and although there were now potentially two JVMs involved, making the name "samevm" somewhat ambiguous, the name of the mode stuck. (Well, the tests themselves were mostly all executing in the "same VM"...)
And along the way, samevm mode has been improved such that it is now the recommended mode to use for all the tests in the OpenJDK langtools repositories. To be fair, making that happen has involved changes to the test harness, the tests and in a couple of cases, to the tools themselves. But the payoffs have been substantial, and we can now execute over 1700 tests in just one sixth of the time it would using using othervm mode. (11 minutes vs. 66 minutes on my laptop)
While we have been able to fix the tests in the langtools repositories to use samevm mode, it has not been so easy to find the resources to do the same for the tests in the jdk repositories. Common problems are tests not closing open files when they exit, and tests trying to set a security manager. These conditions are not an issue when running tests in othervm mode, but both are problematic in samevm mode. If files are left open in the scratch directory, on Windows that will prevent the files being deleted, which in turn will cause problems for all the subsequent tests in the test run. Setting a security manager always been forbidden in samevm because originally it could only be set once anyway, and even now, it is possible to set a security manager that you cannot remove. Together, these and other problems, significantly reduce the number of tests in the jdk repositories that can be run in samevm mode, and that can therefore benefit from the corresponding improvement in performance.
Ideally, it would be possible to make sure that most tests can be run in samevm mode. But updating tests is a risky business at the best of times, and even harder when the original authors are no longer available. And so, in practice, it has been easier to improve the test harness...
The new agentvm mode provides a way to work around these problems. It's still a problem if a test leaves open files when run on Windows, or if it sets a security manager that cannot be unset, but with the new agentvm mode, such problems do not affect the rest of the tests in the test run.
In agentvm mode, jtreg runs in one JVM and creates JVMs with the required characteristics (version of JDK, directory, VM options) as needed to run the tests. More significantly, when each test completes, jtreg will do some housekeeping to restore the JVM involved and the scratch directory to a standard initial state. If any of the housekeeping fails for any reason, the JVMs are allowed to exit. But if the housekeeping is successful, the JVMs are kept available for reuse by any subsequent tests that need JVMs with the same characteristics.
Thus, for "well-behaved" tests, agentvm mode will provide similar performamce to samevm mode, because there will not be any overhead to create a new JVM for each test. But, for less well behaved tests, agentvm mode will automatically degrade to something more like othervm mode, starting new JVMs as necessary for each test.
How many JVMs does jtreg use in agentvm mode at any time? Typically, just two: one to run the test harness, and one to run the tests. That's the same as samevm mode, except it's better, insofar as the JVM used to run the tests will be restarted if there are any problems. There may be three, because agentvm mode allows us to relax the restrictions inherent in samevm mode about not allowing different JVM options or even versions of Java for the compilation and execution of tests. It can do this while still retaining the performance of samevm mode, by creating and using different agent JVMs with the appropriate parameters.
Thus, with agentvm mode, it is now possible to compile tests using a standard JVM but to execute the compiled classes with special VM options, such as for profiling. Previously, this combination was only possible with othervm mode.
And, with agentvm mode, it is now possible to compile tests with a standard version of JDK, and to execute the tests using a version of the Java platform that does not include javac. Previously, jtreg could only test versions of the Java platform that included javac.
Note: if any tests are explicitly marked /othervm, they will obviously be run in their own JVM, separate from the reusable JVMs used for the rest of the tests.
To try out the new agentvm mode, just use the -agentvm option instead of -samevm or -othervm. Or, if you have scripts or Makefiles already set up to run jtreg in samevm mode, there's a backdoor environment variable you can set to use agentvm mode instead, until you're ready to commit any change to your script or Makefile: just set JTREG_USE_AGENTVM_FOR_SAMEVM to true.
Note that when you use agentvm mode, it doesn't make the problems of running a test in samevm mode go away -- it just makes jtreg better able to identify and then tolerate such issues. When issues occur, the performance will degrade to similar to that of othervm mode. To keep the performance up to the level of samevm mode, the issues in the tests reported by jtreg will still need to be fixed. But that's a different story...
There is a new version of jtreg available, 4.1, with a number of useful new features. Most of these are to do with the way that jtreg runs tests, but one feature, limited support for JUnit tests, involves a spec change for the test descriptions, which means that this new version of jtreg will be required to run any testsuites that include such tests.
jtreg can be downloaded from the OpenJDK jtreg page.
Up to now, jtreg has provided two ways of running tests: in "othervm" mode and in "samevm" mode. In "othervm" mode, a new JVM is created for each action that needs it. This provides the maximum isolation between tests, at a very high cost in performance. In "samevm" mode, all Java actions are run in the same VM, which provides high performance at the risk of tests being able to affect the behavior of any subsequent tests. In some cases, a bad test can prevent all subsequent tests from being able to execute.
Problems can arise when setting a security manager such that it cannot be removed when a test completes, or failing to close any open files in the scratch directory, which on Windows means they cannot be deleted before the next test runs.
"Agentvm" mode is a new mode that is "like samevm mode, but better". JVMs created to run tests will be reused for subsequent tests provided that the jtreg harness can successfully perform a certain amount of cleanup after each test. If ever the cleanup fails, the corresponding JVMs will be terminated, and new ones will be started if necessary for any subsequent tests.
In "samevm" mode, because the same JVM is used for both test compilation and test execution, it is not possible to specify JVM options to be used just when executing a test. This restriction is not necessary in "agentvm" mode, and separate JVMs will be created if needed for any actions requiring different JVM characteristics. Assuming the cleanup after a test is successful, all the JVMs used by a test will be saved for reuse by later tests that may need JVMs with those same characteristics. However, if any of the cleanup fails, all the JVMs used by the test will be terminated, because in general it is not possible to determine which of the JVMs may have been at fault.
The new mode can be selected with the -agentvm option. For those situations where "samevm" mode is currently being used, it is also possible to select the new mode by setting the environment variable JTREG_USE_AGENTVM_FOR_SAMEVM to true. This may be convenient if you don't have easy access to the command line used to invoke jtreg.
When jtreg writes HTML report files, it creates links from the report to files in the work directory, and these links could easily be broken if the files in these directories were moved. jtreg now checks if the report directory and work directory are "closely related", meaning they are the same, or one is a parent of the other, or if they have a common parent. In these cases, jtreg will use relative links from the report files to the test result files in the work directory, so that if the two directories are moved together and their relative relationship is retained, the links will remain valid.
Limited support has been added for running JUnit tests. Tests still require a standard jtreg test description, but you can use a new action "@run junit classname", which will invoke the test in the manner of JUnit 4. The class will be compiled if necessary, or it can be compiled explicitly in the standard way with the "@build" and "@compile" actions. (Thanks to John Rose for providing this feature.)
-compilejdk:jdkUp to now, jtreg has only been able to test versions of JDK, or more accurately, versions of Java that include the compiler javac. jtreg now has a new option to specify an alternate JDK to be used for any @compile actions in a test. This includes all implicit @compile actions as well. This means that jtreg is now able to test versions of the Java platform that do not include javac.
Notes:
-compilejdk option is not used, the same version of Java used to compile and run the tests, as now.
-compilejdk is specified, the preference to use a JDK instead of the embedded JRE is no longer required. If in doubt, always do one of the following:
-testjdk for the JDK used to compile and run the tests
-compilejdk for the JDK used to compile the tests,
and -testjdk for the JDK or other version of Java to run the tests
${TESTJAVA}${FS}bin${FS}javac -d classes HelloWorld.java.) jtreg provides a new environment variable, COMPILEJAVA, which will be set to the value of -compilejdk if it was specified, and $TESTJAVA otherwise. If the tests in a testsuite might be run with -compilejdk set, any shell tests using $TESTJAVA to access the compiler should be updated to use $COMPILEJAVA instead.
In various situations, jtreg creates or modifies search paths, such as a classpath or sourcepath, for use in the actions of a test. Previously, jtreg did not check whether the components it was adding to a path actually existed, and this could cause problems for some tests that checked the contents of these paths (for example, with the javac -Xlint or -Xlint:all options.) Components to be added to a path are now checked to verify they identify an existing file or directory.
The @compile action now checks the modification time of source files to be compiled, and gives a warning if the last modified time is in the future.
-allowSetSecurityManager[:value]
jtreg now allows tests in samevm mode and the new agentvm mode to set the security manager. (It was always possible to set the security manager in othervm mode.) If the security manager cannot be reset when the test completes, then in samevm mode, verbose messages will be printed directly to the console output, and in agentvm mode, the test's JVM(s) will be terminated. Acceptable values are yes, on or true and no, off or false. The feature is enabled by default; use -allowSetSecurityManager:off to disable it.
-retainThe -retain option allows you to specify which files in the scratch directory should be retained when a test completes. The default (when the option is not specified) is to leave the files in the scratch directory until the beginning of the next test (if any.) This is convenient when running a single test, but is not convenient when there is a problem cleaning up any files at the beginning of the next test -- by then, the identity of the test that created the files has been lost.
A new suboption is now available for -retain: -retain:none. With this option, all files remaining in the scratch directory when a test completes will be removed immediately. Any problems that may arising in deleting these files will cause the test to be reported with an ERROR result.
-classpathappend in samevm mode
The -classpathappend option did not work in samevm mode, and has now been fixed.
GNU Classpath hackers unite! A collection of diaries by random developers of GNU Classpath related projects (and movie reviews). You can also find some of us in #classpath on irc.gnu.org.
![[RSS 1.0 Feed]](images/rss10.png){kind=small}
![[RSS 2.0 Feed]](images/rss20.png){kind=small}
![[FOAF Subscriptions]](images/foaf.png){kind=small}
![[OPML Subscriptions]](images/opml.png){kind=small}
See also: developer.classpath.org
September 02, 2010 18:12 (UTC)