Planet Classpath

I’ve been following the status of Lisp on ppc64le lately.

I’m running ppc64le Debian sid. Just after I had set up my system, I did some experimentation with what Debian packages had to offer. ECL was the only Lisp that worked, so I started using it for various projects. (I’ve since learned on #sbcl that CLISP built from source is also a good option.)

Ideally I wanted to be able to use SBCL, so I wondered how far into an SBCL bootstrap I could get with ECL as the host compiler. A few months ago, I found the answer was not very far.

Since then though, the SBCL maintainers have been hard at work on two fronts: making SBCL bootstrap against ECL again (on any architecture) and porting SBCL to PPC64.

Recently with a few minor local changes, I was able to bootstrap SBCL natively on my Talos II starting with ECL, then build SBCL again with the bootstrapped SBCL. I had to build ECL from the tip of its development branch, but it has a nice build system and, being based on C, doesn’t require any bootstrapping steps.

So the SBCL ppc64 port is really shaping up; hopefully the next release will advertise ppc64le support. For now, plenty of packages work for me already, like SILME and Quicklisp.

I’d also like to try the same bootstrap procedure on ppc64be ABI version 1 (Debian sid) and ppc64be ABI version 2 (another distro, probably Adélie). I’m working on setting up an environment with qemu-system-ppc64.

Anyway, I’m happy that I can now use SBCL on my Talos II.

elfutils 0.177 was released with various bug fixes (if you ever had issues updating > 2GB ELF files using libelf, this release is for you!) and some new features. One of the features is eu-elfclassify, a utility by Florian Weimer to analyze ELF objects.

People use various tricks to construct ELF files that might make it non-trivial to determine what kind of ELF file you might be dealing with. Even a simple question like “is this a program executable or shared library?” might be tricky given the fact that (static) PIE executables look a lot like shared libraries. And some “shared libraries” are also “program executables”. e.g. Qt likes to provide some information about how the files have been build. So you can link against it as a shared library, but you can also execute it as if it was a program:

$ /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
This is the QtCore library version Qt 5.11.3
(x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 8.3.0)
Installation prefix: /usr
Library path: lib/x86_64-linux-gnu
Include path: include/x86_64-linux-gnu/qt5
Processor features: sse3 sse2[required] ssse3 fma cmpxchg16b sse4.1 sse4.2 movbe popcnt aes avx f16c rdrand bmi avx2 bmi2 rdseed

glibc does the same thing for its shared libraries. Which is nice if you just quickly need to know what libc version is installed on a system, but might make it tricky to determine what kind of ELF file something really is.

eu-classify has a mode that will tell you whether such a file is primarily a shared library or primarily a program executable. And of course is able to classify it as both a library and a program. Hopefully eu-classify can replace the usage of the file (1) utility in various tools, with a more precise way to classify ELF files.

Usage: elfclassify [OPTION...] FILE...
Determine the type of an ELF file.

All of the classification options must apply at the same time to a particular
file.  Classification options can be negated using a "--not-" prefix.

Since modern ELF does not clearly distinguish between programs and dynamic
shared objects, you should normally use either --executable or --shared to
identify the primary purpose of a file.  Only one of the --shared and
--executable checks can pass for a file.

If you want to know whether an ELF object might a program or a shared library
(but could be both), then use --program or --library. Some ELF files will
classify as both a program and a library.

If you just want to know whether an ELF file is loadable (as program or
library) use --loadable.  Note that files that only contain (separate) debug
information (--debug-only) are never --loadable (even though they might contain
program headers).  Linux kernel modules are also not --loadable (in the normal
sense).

Without any of the --print options, the program exits with status 0 if the
requested checks pass for all input files, with 1 if a check fails for any
file, and 2 if there is an environmental issue (such as a file read error or a
memory allocation error).

When printing file names, the program exits with status 0 even if no file names
are printed, and exits with status 2 if there is an environmental issue.

On usage error (e.g. a bad option was given), the program exits with a status
code larger than 2.

The --quiet or -q option suppresses some error warning output, but doesn't
change the exit status.

 Classification options
      --core                 File is an ELF core dump file
      --debug-only           File is a debug only ELF file (separate .debug,
                             .dwo or dwz multi-file)
      --elf                  File looks like an ELF object or archive/static
                             library (default)
      --elf-archive          File is an ELF archive or static library
      --elf-file             File is an regular ELF object (not an
                             archive/static library)
      --executable           File is (primarily) an ELF program executable (not
                             primarily a DSO)
      --library              File is an ELF shared object (DSO) (might also be
                             an executable)
      --linux-kernel-module  File is a linux kernel module
      --loadable             File is a loadable ELF object (program or shared
                             object)
      --program              File is an ELF program executable (might also be a
                             DSO)
      --shared               File is (primarily) an ELF shared object (DSO)
                             (not primarily an executable)
      --unstripped           File is an ELF file with symbol table or .debug_*
                             sections and can be stripped further

 Input flags
  -f, --file                 Only classify regular (not symlink nor special
                             device) files
      --no-stdin             Do not read files from standard input (default)
      --stdin                Also read file names to process from standard
                             input, separated by newlines
      --stdin0               Also read file names to process from standard
                             input, separated by ASCII NUL bytes
  -z, --compressed           Try to open compressed files or embedded (kernel)
                             ELF images

 Output flags
      --matching             If printing file names, print matching files
                             (default)
      --no-print             Do not output file names
      --not-matching         If printing file names, print files that do not
                             match
      --print                Output names of files, separated by newline
      --print0               Output names of files, separated by ASCII NUL

 Additional flags
  -q, --quiet                Suppress some error output (counterpart to
                             --verbose)
  -v, --verbose              Output additional information (can be specified
                             multiple times)

  -?, --help                 Give this help list
      --usage                Give a short usage message
  -V, --version              Print program version

Report bugs to https://sourceware.org/bugzilla.

Compiling with the GCC sanitizers and then fuzzing the resulting binaries might find real bugs. But not all such bugs are security issues. When a CVE is filed there is some pressure to treat such an issue with urgency and push out a fix as soon as possible. But taking your time and making sure an issue can be replicated/exploited without the binary being instrumented by the sanitizer is often better.

This was the case for CVE-2019-12900BZ2_decompress in decompress.c in bzip2 through 1.0.6 has an out-of-bounds write when there are many selectors“.

The bzip2 project had lost the domain which it had used for the last 15 years. And it hadn’t seen an official release since 2010. The bzip2 project homepage, documentation and downloads had already been moved back to sourceware.org. And a new bug tracker, development mailinglist and git repository had been setup. But we were still in the middle of a code cleanup (removing references to the old homepage, updating the manual and adding various cleanups that distros had made to the code) when the CVE was filed.

The issue reported was discovered by a fuzzer ran against a bzip2 binary compiled with gcc -fsanitizer=undefined. Which produced the following error:

decompress.c:299:10: runtime error: index 18002 out of bounds for type 'UChar [18002]'

The DState struct given to the BZ2_decompress function has a field defined as UChar selectorMtf[BZ_MAX_SELECTORS]; where BZ_MAX_SELECTORS is 18002. So the patch that came with the security report looked totally reasonable.

--- a/decompress.c
+++ b/decompress.c
@@ -284,15 +284,15 @@ Int32 BZ2_decompress ( DState* s )
284      /*--- Now the selectors ---*/
285      GET_BITS(BZ_X_SELECTOR_1, nGroups, 3);
286      if (nGroups < 2 || nGroups > 6) RETURN(BZ_DATA_ERROR);
287      GET_BITS(BZ_X_SELECTOR_2, nSelectors, 15);
288 -    if (nSelectors < 1) RETURN(BZ_DATA_ERROR);
    +    if (nSelectors < 1 || nSelectors > BZ_MAX_SELECTORS) RETURN(BZ_DATA_ERROR);
289      for (i = 0; i < nSelectors; i++) {
290         j = 0;
293         while (True) {
294            GET_BIT(BZ_X_SELECTOR_3, uc);
295            if (uc == 0) break;
296            j++;
297            if (j >= nGroups) RETURN(BZ_DATA_ERROR);
298         }
299         s->selectorMtf[i] = j; /* array overrun! */
300      }

Without the new nSelectors > BZ_MAX_SELECTORS guard the code could write beyond the selectorMtf array, which is undefined behavior. The undefined behavior in this case would be writing to memory addresses after the array. Given that an attacker could define nSelectors as big as they want, they would be able to override any memory after the array. This seemed urgent enough to do a new release quickly with this fix.

bzip2 1.0.7 was released. But the next day we already got bug reports that the fix broke decompression of some existing .bz2 files. This didn’t really make sense at first. BZ_MAX_SELECTORS was the theoretical maximum number of selectors that could validly be used in a .bz2 file. But some testing did confirm that these files did define a handful more selectors than were actually used. It turned out that some alternative bzip2 implementations used a slightly bigger maximum for the number of selectors (rounded up to a factor 8) which they might define, but didn’t expect to be used.

Julian Seward came up with a fix that split the max number of selectors in two. The original theoretical max that bzip2 would encode, and a bigger (rounded up to a factor 8) max that would be accepted when decompressing. This seemed to fix the issue for real, while still accepting some slightly “wrong” .bz2 files. The original code had worked for these because the array overwrite was only a few bytes, and the DState struct has extra state right after the selectorMtf array. The UChar len[BZ_N_GROUPS][BZ_MAX_ALPHA_SIZE] array (6 * 258 = 6192 bytes), which was only written to after the selectors were read. So the memory overwrite was almost immediately corrected and didn’t do any harm because it was just such a small amount. The new code would still protect against real “too bignSelector values.

But we still didn’t feel completely confident we had fixed things correctly. One issue was that bzip2 never had a really good testsuite. Testing was mostly done ad-hoc by developers on a random collection of .bz2 files that they happened to have around. Luckily some alternative bzip2 implementations had created more formal testsuites. The .bz2 testfiles of those projects were collected and a testframe was created that ran bzip2 on both correct and known bad .bz2 files (optionally using valgrind to catch bad memory usage). This was a really good thing. The testsuite was added to the bzip2 buildbot. Which immediately flagged one testcase (32767.bz2) as BAD!

The 32767.bz2 testcase has the max number of selectors that the file format allows (2^15 - 1 = 32767). The .bz2 file format reserves 15 bits for the number of selectors used in a block. This is because to express the max of 18002 selectors can only be expressed when using 15 bits. That testcase could be decompressed correctly by bzip2 1.0.6 (or earlier), but not by the new bzip2 version that checked the number of selectors was “sane“. When the original bzip2 1.0.6 code was compiled with gcc -fsanitize=undefined the selectorMtf array overwrite was (correctly) reported. But surprisingly when ran under valgrind memcheck no bad memory usage was reported.

Some more investigation revealed that although this was an example of the most extreme possible selectorMtf array overwrite, it still only wrote over already allocated memory and that memory was not used before being assigned correct values. The selectorMtf array could hold 18002 bytes. 32767 – 18002 = 14765 bytes that could be overwritten after the array. But the DState struct had 3 more arrays after the selectorMtf and len arrays. Each defined as UInt32 [BZ_N_GROUPS][BZ_MAX_ALPHA_SIZE], which is 3 * 4 * 6 * 258 = 18576 bytes. And all state after the selectorMtf array in the DState struct would be assigned values right after reading the selectors. And none of the excess selector values would ever be used. So even though there really was an array overwrite, it was completely harmless!

That knowledge allowed us to write a much simpler patch that just skipped over the extra selectors without storing them. And release bzip2 1.0.8 that decompressed all the same files that 1.0.6 and earlier could.

In the end it was good for the bzip2 project to have a bit of an emergency. It brought people together who cared deeply about making sure bzip2 survives as a project, it got us automated release scripts, a new testsuite, buildbots, various other fixes upstreamed from distros and bzip2 is now part of oss fuzz (so we might get earlier warnings about similar issues in the future) and there is now a kind of roadmap for how to move forward

But part of the panic was also completely unnecessary. Yes, there was a way to trigger undefined behavior, but with any current compiler that behavior was actually defined, it would write over known (bounded) memory, memory that otherwise was correctly used and defined. We should have insisted on having a real reproducer, that could be triggered under valgrind memcheck. The instrumentation of the undefined sanitizer was not enough to show a real issue. We were lucky, it could certainly have been, or become, a real issue if the DState structure layout would have been different, if some constants were larger or smaller or if the compiler was smarter (it could have decided that writing after the array could never happen and so “optimize” the program assuming some loops were bounded). So fixing the bug was certainly the right thing to do. But in practice it never was a real security issue and we placed too much value in the fact that a CVE was assigned to it.

In this miniseries, I’d like to introduce a couple of new developments of the Shenandoah GC that are upcoming in JDK 13. This here is about a new architecture and a new operating system that Shenandoah will be working with.

Solaris

Only about a few days ago, Bellsoft contributed a change that allowed Shenandoah to build and run on Solaris. Shenandoah itself has zero operating-system-specific code in it, and is therefore relatively easy to port to new operating systems. In this case, it mostly amounts to a batch of fixes to make the Solaris compiler happy, like removing a trailing comma in enums.

One notable gotcha that we hit was with Solaris 10. Contrary to what later versions of Solaris do, and what basically all relevant other operating systems do, Solaris 10 maps user memory to upper address ranges, e.g. to addresses starting with 0xff… instead of 0x7f. Other operating systems reserve the upper half of the address space to kernel memory. This conflicted with an optimization of Shenandoah’s task queues, which would encode pointers assuming it has some spare space in the upper address range. It was easy enough to disable via build-time-flag, and so Aleksey did. The fix is totally internal to Shenandoah GC and does not affect the representation of Java references in heap. With this change, Shenandoah can be built and run on Solaris 10 and newer (and possibly older, but we haven’t tried). This is not only interesting for folks who want Shenandoah to run on Solaris, but also for us, because it requires the extra bit of cleanliness to make non-mainline toolchains happy.

The changes for Solaris support are already in JDK 13 development repositories, and are in-fact already backported to Shenandoah’s JDK 11 and JDK 8 backports repositories.

x86_32

Shenandoah used to support x86_32 in “passive” mode long time ago. This mode relies only on stop-the-world GC to avoid implementing barriers (basically, runs Degenerated GC all the time). It was an interesting mode to see the footprint numbers you can get with uncommits and slimmer native pointers with really small microservice-size VMs. This mode was dropped before integration upstream, because many Shenandoah tests expect all heuristics/modes to work properly, and having the rudimentary x86_32 support was breaking tier1 tests. So we disabled it.

Today, we have significantly simplified runtime interface thanks to load-reference-barriers and elimination of separate forwarding pointer slot, and we can build the fully concurrent x86_32 on top of that. This allows us to maintain 32-bit cleanness in Shenandoah code (we have fixed >5 bugs ahead of this change!), plus serves as proof of concept that Shenandoah can be implemented on 32-bit platforms. It is interesting in scenarios where the extra footprint savings are important like in containers or embedded systems. The combination of LRB+no more forwarding pointer+32bit support gives us the current lowest bounds for footprint that would be possible with Shenandoah.

The changes for x86_32 bit support are done and ready to be integrated into JDK 13. However, they are currently waiting for the elimination-of-forwarding-pointer change, which in turn is waiting for a nasty C2 bug fix. The plan is to later backport it to Shenandoah JDK 11 and JDK 8 backports – after load-reference-barriers and elimination-of-forwarding-pointer changes have been backported.

Other arches and OSes

With those two additions to OS and architecturs support, Shenandoah will soon be available (e.g. known to build and run) on four operating systems: Linux, Windows, MacOS and Solaris, plus 3 architectures: x86_64, arm64 and x86_32. Given Shenandoah’s design with zero OS specific code, and not overly complex architecture-specific code, we may be looking at more OSes or architectures to join the flock in future releases, if anybody finds it interesting enough to implement.

As always, if you don’t want to wait for releases, you can already have everything and help sort out problems: check out The Shenandoah GC Wiki.

In this miniseries, I’d like to introduce a couple of new developments of the Shenandoah GC that are upcoming in JDK 13. The change I want to talk about here addresses another very frequent, perhaps *the* most frequent concern about Shenandoah GC: the need for an extra word per object. Many believe this is a core requirement for Shenandoah, but it is actually not, as you would see below.

Let’s first look at the usual object layout of an object in the Hotspot JVM:

 0: [mark-word  ]
 8: [class-word ]
16: [field 1    ]
24: [field 2    ]
32: [field 3    ]

Each section here marks a heap-word. That would be 64 bits on 64 bit architectures and 32 bits on 32 bit architectures.

The first word is the so-called mark-word, or header of the object. It is used for a variety of purposes: it can keep the hash-code of an object, it has 3 bits that are used for various locking states, some GCs use it to track object age and marking status, and it can be ‘overlaid’ with a pointer to the ‘displaced’ mark, to an ‘inflated’ lock or, during GC, the forwarding pointer.

The second word is reserved for the klass-pointer. This is simply a pointer to the Hotspot-internal data-structure that represents the class of the object.

Arrays would have an additional word next to store the arraylength. What follows afterwards is the actual ‘payload’ of the object, i.e. fields and array elements.

When running with Shenandoah enabled, the layout would look like this instead:

-8: [fwd pointer]
 0: [mark-word  ]
 8: [class-word ]
16: [field 1    ]
24: [field 2    ]
32: [field 3    ]

The forward pointer is used for Shenandoah’s concurrent evacuation protocol:

  • Normally it points to itself -> the object is not evacuated yet
  • When evacuating (by the GC or via a write-barrier), we first copy the object, then install new forwarding pointer to that copy using an atomic compare-and-swap, possibly yielding a pointer to an offending copy. Only one copy wins.
  • Now, the canonical copy to read-from or write-to can be found simply by reading this forwarding pointer.

The advantage of this protocol is that it’s simple and cheap. The cheap aspect is important here, because, remember, Shenandoah needs to resolve the forwardee for every single read or write, even primitive ones. And using this protocol, the read-barrier for this would be a single instruction:

mov %rax, (%rax, -8)

That’s about as simple as it gets.

The disadvantage is obviously that it requires more memory. In the worst case, for objects without any payload, one more word for otherwise two-word object. That’s 50% more. With more realistic object size distributions, you’d still end up with 5%-10% more overhead, YMMV. This also results in reduced performance: allocating the same number of objects would hit the ceiling faster than without that overhead, prompting GCs more often, and therefore reduce throughput.

If you’ve read the above paragraphs carefully, you will have noticed that the mark-word is also used/overlaid by some GCs to carry the forwarding pointer. So why not do the same in Shenandoah? The answer is (or used to be), that reading the forwarding pointer requires a little more work. We need to somehow distinguish a true mark-word from a forwarding pointer. That is done by setting the lowest two bits in the mark-word. Those are usually used as locking bits, but the combination 0b11 is not a legal combination of lock bits. In other words: when they are set, the mark-word, with the lowest bits masked to 0, is to be interpreted as forwarding pointer. This decoding of the mark word is significantly more complex than the above simple read of the forwarding pointer. I did in-fact build a prototype a while ago, and the additional cost of the read-barriers was prohibitive and did not justify the savings.

All of this changed with the recent arrival of load reference barriers:

  • We no longer require read-barriers, especially not on (very frequent) primitive reads
  • The load-reference-barriers are conditional, which means their slow-path (actual resolution) is only activated when 1. GC is active and 2. the object in question is in the collection set. This is fairly infrequent. Compare that to the previous read-barriers which would be always-on.
  • We no longer allow any access to from-space copies. The strong invariant guarantees us that we only ever read from and write to to-space copies.

Two consequences of these are: the from-space copy is not actually used for anything, we can use that space to put the forwarding pointer, instead of reserving an extra word for it. We can basically nuke the whole contents of the from-space copy, and put the forwarding pointer anywhere. We only need to be able to distinguish between ‘not forwarded’ (and we don’t care about other contents) and ‘forwarded’ (the rest is forwarding pointer).

It also means that the actual mid- and slow-paths of the load-reference-barriers are not all that hot, and we can easily afford to do a little bit of decoding there. It amounts to something like (in pseudocode):

oop decode_forwarding(oop obj) {
  mark m = obj->load_mark();
  if ((m & 0b11) == 0b11) {
    return (oop) (m & ~0b11);
  } else {
    return obj;
  }
}

While this looks noticably more complicated than the above simple load of the forwarding pointer, it is still basically a free lunch because it’s only ever executed in the not-very-hot mid-path of the load-reference-barrier. With this, the new object layout would be:

  0: [mark word (or fwd pointer)]
  8: [class word]
 16: [field 1]
 24: [field 2]
 32: [field 3]

Doing so has a number of advantages:

  • Obviously, it reduces Shenandoah’s memory footprint by putting away with the extra word.
  • Not quite as obviously, this results in increased throughput: we can now allocate more objects before hitting the GC trigger, resulting in fewer cycles spent in actual GC.
  • Objects are packed more tightly, which results in improved CPU cache pressure.
  • Again, the required GC interfaces are simpler: where we needed special implementations of the allocation paths (to reserve and initialize the extra word), we can now use the same allocation code as any other GC

To give you an idea of the throughput improvements: all the GC sensitive benchmarks that I have tried showed gains between 10% and 15%. Others benefited less or not at all, but that is not surprising for benchmarks that don’t do any GC at all. But it is important to note that the extra decoding cost does not actually show up anywhere, it is basically negligible. It probably would show up on heavily evacuating workloads. But most applications don’t evacuate that much, and most of the work is done by GC threads anyway, making midpath decoding cheap enough.

The implementation of this has recently been pushed to the shenandoah/jdk repository. We are currently shaking out one last known bug, and then it’s ready to go upstream into JDK 13 repository. The plan is to eventually backport it to Shenandoah’s JDK 11 and JDK 8 backports repositories, and from there into RPMs. If you don’t want to wait, you can already have it: check out The Shenandoah GC Wiki.

The past release of 27.9.15 ArcticFox has the Developer Tools working again, they were broken previously because of excessive work on Private browsing.

You can see them here in full action:


ArcticFox continues the work as a fork of PaleMoon trying to catch up with past releases of FireFox.
It has been succesfully backported up to MacOS 10.6 SnowLeopard, is working reliably on Linux x86, amd64 and PowerPC 32bit and 64bit.

If you like the browser, we need your help!
My white MacBook is a perfectly fine computer, has an excellent screen and keyboard (superior to later models I have) so even if it is running a legacy OS version, I'd love to continue using it and, perhaps, other of you are in the same situation.

Besides ArcticFox (which I got running on 10.6, but not on 10.5 yet) and various developer tools, the most essential tool I like to have is GIMP, also given the excellent LCD this Laptop has.

Some work from macports to get up-to-date tools and dependent libraries, I patched its package.

Little work was needed in GIMP itself, I shared all the patches upstream and I hope they will be accepted. The remaining issues are in GEGL, but different solutions are available: essentially realpath doesn't like a null argument.

Enjoy this screenshot as a proof.



Next question: will it work on PowerPC too? I hope so! Getting GIMP to work on 10.5 opens this possibility.

P.S.: this post was written directly on the MacBook i386 using TenFourFox running natively on Intel!

The IcedTea project provides a harness to build the source code from OpenJDK using Free Software build tools, along with additional features such as the ability to build against system libraries and support for alternative virtual machines and architectures beyond those supported by OpenJDK.

This release updates our OpenJDK 7 support in the 2.6.x series with the October 2018 security fixes from OpenJDK 7 u201.

If you find an issue with the release, please report it to our bug database under the appropriate component. Development discussion takes place on the distro-pkg-dev OpenJDK mailing list and patches are always welcome.

Full details of the release can be found below.

What’s New?

New in release 2.6.16 (2019-01-01)

  • Security fixes
  • Import of OpenJDK 7 u201 build 0
    • S7058700: Unexpected exceptions and timeouts in SF2 parser code
    • S7098755: test/sun/misc/JarIndex/metaInfFilenames/Basic.java should use supported compiler interface
    • S7104650: rawtype warnings in several net, nio and security source files
    • S7116722: Miscellaneous warnings sun.misc ( and related classes )
    • S7117249: fix warnings in java.util.jar, .logging, .prefs, .zip
    • S7142888: sun/security/tools/jarsigner/ec.sh fail on sparc
    • S8044860: Vectors and fixed length fields should be verified for allowed sizes.
    • S8049834: Two security tools tests do not run with only JRE
    • S8054431: Some of the input validation in the javasound is too strict
    • S8074462: Handshake messages can be strictly ordered
    • S8130132: jarsigner should emit warning if weak algorithms or keysizes are used
    • S8142927: Feed some text to STDIN in ProcessTools.executeProcess()
    • S8146377: test/sun/security/tools/jarsigner/concise_jarsigner.sh failing
    • S8158887: sun/security/tools/jarsigner/concise_jarsigner.sh timed out
    • S8164480: Crash with assert(handler_address == SharedRuntime::compute_compiled_exc_handler(..) failed: Must be the same
    • S8168405: Pending exceptions in java.base/windows/native
    • S8172529: Use PKIXValidator in jarsigner
    • S8180289: jarsigner treats timestamped signed jar invalid after the signer cert expires
    • S8190674: sun/security/tools/jarsigner/TimestampCheck.java failed with java.nio.file.NoSuchFileException: ts2.cert
    • S8193892: Impact of noncloneable MessageDigest implementation
    • S8204667: Resources not freed on exception
    • S8207336: Build failure in JDK8u on Windows after fix 8207260
    • S8208350: Disable all DES cipher suites
    • S8208660: JDK 8u191 l10n resource file update
    • S8208754: The fix for JDK-8194534 needs updates
    • S8211107: LDAPS communication failure with jdk 1.8.0_181
    • S8211731: Reconsider default option for ClassPathURLCheck change done in JDK-8195874

The tarballs can be downloaded from:

We provide both gzip and xz tarballs, so that those who are able to make use of the smaller tarball produced by xz may do so.

The tarballs are accompanied by digital signatures available at:

These are produced using my public key. See details below.

  • PGP Key: ed25519/0xCFDA0F9B35964222 (hkp://keys.gnupg.net)
  • Fingerprint = 5132 579D D154 0ED2 3E04 C5A0 CFDA 0F9B 3596 4222

GnuPG >= 2.1 is required to be able to handle this key.

SHA256 checksums:

  • f6bf1388d3dc6f7206f49702a00f2836f11841987d74a976b315843959818213 icedtea-2.6.16.tar.gz
  • 64b4d156d0a1b253a3df90092ccf5605f81a5d0300434b5fd19444c7a9245585 icedtea-2.6.16.tar.gz.sig
  • 6c670e75549dfd4df63a4a36636c13a5040231e7f8601f9d43bf875589df7b69 icedtea-2.6.16.tar.xz
  • 0a4a0f95ecbf34302e4368b4f71a51a0da059a2a0839f44919353ae6a67f3acb icedtea-2.6.16.tar.xz.sig

The checksums can be downloaded from:

A 2.6.16 ebuild for Gentoo is available.

The following people helped with these releases:

We would also like to thank the bug reporters and testers!

To get started:

$ tar xzf icedtea-2.6.16.tar.gz

or:

$ tar x -I xz -f icedtea-2.6.16.tar.xz

then:

$ mkdir icedtea-build
$ cd icedtea-build
$ ../icedtea-2.6.16/configure
$ make

Full build requirements and instructions are available in the INSTALL file.

Happy hacking!

The IcedTea project provides a harness to build the source code from OpenJDK using Free Software build tools, along with additional features such as the ability to build against system libraries and support for alternative virtual machines and architectures beyond those supported by OpenJDK.

This release updates our OpenJDK 7 support in the 2.6.x series with the July 2018 security fixes from OpenJDK 7 u191.

If you find an issue with the release, please report it to our bug database under the appropriate component. Development discussion takes place on the distro-pkg-dev OpenJDK mailing list and patches are always welcome.

Full details of the release can be found below.

What’s New?

New in release 2.6.15 (2018-12-31)

  • Security fixes
  • New features
    • PR3629: Install symlinks to tapsets in SystemTap directory
    • PR3657: Sync desktop files with Fedora/RHEL versions again
    • PR3659: Support RHEL multilib installations which use the /usr/lib/jvm/java-1.x.0-openjdk.${arch} naming
  • Import of OpenJDK 7 u191 build 0
    • S8005661: [parfait] Possible buffer overrun in jdk/src/solaris/native/sun/awt/awt_GraphicsEnv.c
    • S8005695: [parfait] Format string argument mismatch in jdk/src/solaris/native/sun/xawt/XToolkit.c
    • S8005752: [parfait] False positive function call mismatch at jdk/src/solaris/native/sun/xawt/XWindow.c
    • S8034856: gcc warnings compiling src/solaris/native/sun/security/pkcs11
    • S8034857: gcc warnings compiling src/solaris/native/sun/management
    • S8035054: JarFacade.c should not include ctype.h
    • S8035287: gcc warnings compiling various libraries files
    • S8203182: Release session if initialization of SunPKCS11 Signature fails
    • S8205491: adjust reflective access checks
    • S8205587, PR3606: Implicit function declaration in jni_util.c
  • Import of OpenJDK 7 u191 build 1
    • S8051972: sun/security/pkcs11/ec/ReadCertificates.java fails intermittently
    • S8076117: EndEntityChecker should not process custom extensions after PKIX validation
    • S8157898: SupportedDSAParamGen.java failed with timeout
    • S8170035: When determining the ciphersuite lists, there is no debug output for disabled suites.
    • S8176183: sun/security/mscapi/SignedObjectChain.java fails on Windows
    • S8187635: On Windows Swing changes keyboard layout on a window activation
    • S8196224: Even better Internet address support
    • S8196854: TestFlushableGZIPOutputStream failing with IndexOutOfBoundsException
    • S8197943: Unable to use JDWP API in JDK 8 to debug JDK 9 VM
    • S8200359: (tz) Upgrade time-zone data to tzdata2018d
    • S8201433: Fix potential crash in BufImg_SetupICM
    • S8202585: JDK 8u181 l10n resource file update
    • S8202996: Remove debug print statements from RMI fix
    • S8203233: (tz) Upgrade time-zone data to tzdata2018e
    • S8203368: ObjectInputStream filterCheck method throws NullPointerException
  • Import of OpenJDK 7 u191 build 2
    • S8207151, PR3604: Frequent JVM Crash SIGSEGV same stacktrace location during tomcat start with hibernate init on 7u181-2.6.14-0ubuntu0.1
  • Backports
    • S8075942, PR3605: ArrayIndexOutOfBoundsException in sun.java2d.pisces.Dasher.goTo
  • Bug fixes
    • PR3616: Don’t include timestamps in generated documentation
    • PR3631: Use ${datadir} when specifying default tz.properties location
    • PR3652: Detect whether -Xprefer:source and -J-Xmx<limit> can be used, rather than assuming
    • PR3663: IcedTea installing symlinks to SystemTap directory rather than individual tapsets
  • SystemTap
    • PR3633: arc_priority representation creates an implicit limit on character sequence within regexp
  • AArch64 port
    • S8207345, PR3614: Trampoline generation code reads from uninitialized memory
    • PR3615: Fix whitespace in hotspot/src/cpu/aarch64

The tarballs can be downloaded from:

We provide both gzip and xz tarballs, so that those who are able to make use of the smaller tarball produced by xz may do so.

The tarballs are accompanied by digital signatures available at:

These are produced using my public key. See details below.

  • PGP Key: ed25519/0xCFDA0F9B35964222 (hkp://keys.gnupg.net)
  • Fingerprint = 5132 579D D154 0ED2 3E04 C5A0 CFDA 0F9B 3596 4222

GnuPG >= 2.1 is required to be able to handle this key.

SHA256 checksums:

  • 6275b9cb67e82726ddd12ff52103a9114d7d2998a0c21a8024b46607112a86b9 icedtea-2.6.15.tar.gz
  • 64273b5a8c5136368e4f5e0e525f17472684a4c07b54d20e2983923a874d1955 icedtea-2.6.15.tar.gz.sig
  • fea1f128b09fe18249223e9efe2f8f95af886a22c26e1e8593068dc3dd51a5b9 icedtea-2.6.15.tar.xz
  • bf814a0f15a9bd94c96639a3fdc3303f672b3f6f031c04107745b76ddb4acffd icedtea-2.6.15.tar.xz.sig

The checksums can be downloaded from:

A 2.6.15 ebuild for Gentoo is available.

The following people helped with these releases:

We would also like to thank the bug reporters and testers!

To get started:

$ tar xzf icedtea-2.6.15.tar.gz

or:

$ tar x -I xz -f icedtea-2.6.15.tar.xz

then:

$ mkdir icedtea-build
$ cd icedtea-build
$ ../icedtea-2.6.15/configure
$ make

Full build requirements and instructions are available in the INSTALL file.

Happy hacking!

I finished building my Talos II system, and I decided to post my thoughts on it here.

This is an excellent machine, the best workstation I’ve ever used. It’s very fast — it compiles Emacs and the kernel in 2 minutes, and just spins up the fans a little bit while doing so (under normal operation it’s quiet). And it’s in a completely different league in terms of openness than any other computer I’ve owned (and any other high performance computer on the market today).

The system came with a nice user’s manual, and full schematics! I’ve already referred to the schematics to set up the pinmux for a rarely-used serial port so that I could use a TTL serial cable I had lying around (I submitted the device tree patch to Raptor).

It’s really two computers in one: the “baseboard management controller” (BMC), a low power ARMv6, with a full distro on it, and the main POWER9 CPUs. The BMC boots up as soon as you plug in the power supply, before you even press the power button. (It would be nice if there were an ncurses top-like viewer for fan speeds and temperatures that I could leave running on the BMC serial console, but I haven’t found such a thing yet.)

It has serial ports everywhere! One right into the main CPU, and two into the BMC. This is great for low level development, e.g., breaking into bootcode at an early stage.

I left the machine running overnight after first booting it. My neighbourhood had a power glitch, and in the morning I discovered the main CPU was off, and power cycling it via the BMC wasn’t working. Before unplugging and plugging back in, I asked on #talos-workstation, and it turns out I had hit a bug in the first-release firmware. With special commands I was able to power cycle the main CPU just via BMC software (no unplug required). Wanting to know the details, I asked if there was a data sheet for the chip I was interacting with. The amazing thing, from an openness perspective, is that one of the Raptor engineers instead referred me directly to the Verilog source code of the FPGA handling the power sequencing. No searching for datasheets, no black box testing, just straight source code (which is recompilable using an open source FPGA toolchain, BTW.)

It’s so refreshing to not have to do reverse engineering and speculation on opaque things when something fails. On this machine, everything is there, you just look up the source code.

I’ve always disliked the inflexibility and opacity of BIOS/EFI in the x86 world. IBM has done an amazing job here with OpenPOWER. All the early boot code is open, no locked management engines or anything like that. And they’ve adopted Petitboot as the bootloader. It runs on a Linux kernel, so I was able to bootstrap via deboostrap over NFS by building everything within the bootloader environment. Running a compiler in a boot environment is surreal. Even with free options on x86 like libreboot or coreboot, and GRUB, the boot environment is extremely limited. With Petitboot at times I wondered if I even needed to boot into a “desktop” kernel (at least for serial-only activities.)

Now I’m setting up my development environment and I’m learning about the PPC64 ELFv2 ABI, with a view toward figuring out how to bootstrap SBCL. I feel lucky that I got a POWER9 machine early while there are still some rough edges to figure out.

One of my hobbies in GDB is cleaning things up. A lot of this is modernizing and C++-ifying the code, but I’ve also enabled a number of warnings and other forms of code checking in the last year or two. I thought it might be interesting to look at the impact, on GDB, of these things.

So, I went through my old warning and sanitizer patch series (some of which are still in progress) to see how many bugs were caught.

This list is sorted by least effective first, with caveats.

-fsanitize=undefined; Score: 0 or 10

You can use -fsanitize=undefined when compiling to have GCC detect undefined behavior in your code.  This series hasn’t landed yet (it is pending some documentation updates).

We have a caveat already!  It’s not completely fair to put UBsan at the top of the list — the point of this is that it detects situations where the compiler might do something bad.  As far as I know, none of the undefined behavior that was fixed in this series caused any visible problem (so from this point of view the score is zero); however, who knows what future compilers might do (and from this point of view it found 10 bugs).  So maybe UBSan should be last on the list.

Most of the bugs found were due to integer overflow, for example decoding ULEB128 in a signed type.  There were also a couple cases of passing NULL to memcpy with a length of 0, which is undefined but should probably just be changed in the standard.

-Wsuggest-override; Score: 0

This warning will fire if you have a method that could have been marked override, but was not.  This did not catch any gdb bugs.  It does still have value, like everything on this list, because it may prevent a future bug.

-Wduplicated-cond; Score: 1

This warning detects duplicated conditions in an if-else chain.  Normally, I suppose, these would arise from typos or copy/paste in similar conditions.  The one bug this caught in GDB was of that form — two identical conditions in an instruction decoder.

GCC has a related -Wduplicated-branches warning, which warns when the arms of an if have identical code; but it turns out that there are some macro expansions in one of GDB’s supporting libraries where this triggers, but where the code is in fact ok.

-Wunused-variable; Score: 2

When I added this warning to the build, I thought the impact would be removing some dead code, and perhaps a bit of fiddling with #ifs.  However, it caught a couple of real bugs: cases where a variable was unused, but should have been used.

-D_GLIBCXX_DEBUG; Score: 2

libstdc++ has a debug mode that enables extra checking in various parts of the C++ library.  For example, enabling this will check the irreflexivity rule for operator<.  While the patch to enable this still hasn’t gone in — I think, actually, it is still pending some failure investigation on some builds — enabling the flag locally has caught a couple of bugs.  The fixes for these went in.

-Wimplicit-fallthrough; Score: 3

C made a bad choice in allowing switch cases to fall through by default.  This warning rectifies this old error by requiring you to explicitly mark fall-through cases.

Apparently I tried this twice; the first time didn’t detect any bugs, but the second time — and I don’t recall what, if anything, changed — this warning found three bugs: a missing break in the process recording code, and two in MI.

-Wshadow=local; Score: 3

Shadowing is when a variable in some inner scope has the same name as a variable in an outer scope.  Often this is harmless, but sometimes it is confusing, and sometimes actively bad.

For a long time, enabling a warning in this area was controversial in GDB, because GCC didn’t offer enough control over exactly when to warn, the canonical example being that GCC would warn about a local variable named “index“, which shadowed a deprecated C library function.

However, now GCC can warn about shadowing within a single function; so I wrote a series (still not checked in) to add -Wshadow=local.

This found three bugs.  One of the bugs was found by happenstance: it was in the vicinity of an otherwise innocuous shadowing problem.  The other two bugs were cases where the shadowing variable caused incorrect behavior, and removing the inner declaration was enough to fix the problem.

-fsanitize=address; Score: 6

The address sanitizer checks various typical memory-related errors: buffer overflows, use-after-free, and the like.  This series has not yet landed (I haven’t even written the final fix yet), but meanwhile it has found 6 bugs in GDB.

Conclusion

I’m generally a fan of turning on warnings, provided that they rarely have false positives.

There’s been a one-time cost for most warnings — a lot of grunge work to fix up all the obvious spots.  Once that is done, though, the cost seems small: GDB enables warnings by default when built from git (not when built from a release), and most regular developers use GCC, so build failures are caught quickly.

The main surprise for me is how few bugs were caught.  I suppose this is partly because the analysis done for new warnings is pretty shallow.  In cases like the address sanitizer, more bugs were found; but at the same time there have already been passes done over GDB using Valgrind and memcheck, so perhaps the number of such bugs was already on the low side.

I’ve been working a bit more on my Emacs JIT, in particular on improving function calling.  This has been a fun project so I thought I’d talk about it a bit.

Background

Under the hood, the Emacs Lisp implementation has a few different ways to call functions.  Calls to or from Lisp are dispatched depending on what is being called:

  • For an interpreted function, the arguments are bound and then the interpreter is called;
  • For a byte-compiled function using dynamic binding, the arguments are bound and then the bytecode interpreter is called;
  • For a byte-compiled function using lexical binding, an array of arguments is passed to the bytecode interpreter;
  • For a function implemented in C (called a “subr” internally), up to 8 arguments are supported directly — as in, C functions of the form f(arg,arg,...); for more than that, an array of arguments is passed and the function itself must decide which slot means what.  That is, there are exactly 10 forms of subr (actually there are 11 but the one missing from this description is used for special forms, which we don’t need to think about here).

Oh, let’s just show the definition so you can read for yourself:

union {
Lisp_Object (*a0) (void);
Lisp_Object (*a1) (Lisp_Object);
Lisp_Object (*a2) (Lisp_Object, Lisp_Object);
Lisp_Object (*a3) (Lisp_Object, Lisp_Object, Lisp_Object);
Lisp_Object (*a4) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
Lisp_Object (*a5) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
Lisp_Object (*a6) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
Lisp_Object (*a7) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
Lisp_Object (*a8) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
Lisp_Object (*aUNEVALLED) (Lisp_Object args);
Lisp_Object (*aMANY) (ptrdiff_t, Lisp_Object *);
} function;

Initial Approach

Initially the JIT worked like a lexically-bound bytecode function: an array of arguments was passed to the JIT-compiled function.  The JIT compiler emitted a bunch of code to decode the arguments.

For Lisp functions taking a fixed number of arguments, this wasn’t too bad — just moving values from fixed slots in the array to fixed values in the IR.

Handling optional arguments was a bit uglier, involving a series of checks and branches, so that un-bound arguments could correctly be set to nil.  These were done something like:

if nargs < 1 goto nope1
arg0 = array[0]
if nargs < 2 goto nope2
arg1 = array[1]
goto first_bytecode
nope1: arg0 = nil
nope2: arg1 = nil
first_bytecode: ...

&rest arguments were even a bit worse, requiring a call to create a list.  (This, I think, can’t be avoided without a much smarter compiler, one that would notice when reifying the list could be avoided.)

Note that calling also has to use the fully generic approach: we make a temporary array of arguments, then call a C function (Ffuncall) that does the dispatching to the callee.  This is also a source of inefficiency.

Today

Recently, I changed the JIT from this approach to use the equivalent of the subr calling convention.  Now, any function with 8 or fewer (non-&rest) arguments is simply an ordinary function of N arguments, and we let the already-existing C code deal with optional arguments.

Although this often makes the generated assembly simpler, it won’t actually perform any better — the same work is still being done, just somewhere else.  However, this approach does use a bit less memory (most JIT-compiled functions are shorter); and it opens the door to an even bigger improvement.

The Future

What I’m implementing now is an approach to removing most of the overhead from JIT-compiled function calls.

Now, ideally what I’d like is to have every call site work “like C”: move the arguments to exactly where the callee expects them to be, and then call.  However, while looking at this I found some problems that make it tricky:

  • We still need to be able to call Lisp functions from C, so we’re limited to, at best, subr-style calling conventions;
  • While &rest arguments are straightforward (in our simple compiler, somebody just has to make the list); &optional arguments don’t have a good C-like analog.  The callee could push extra arguments, but…
  • In Lisp, a function can be redefined at any time, and it is fine to change the function’s signature.

Consider this example:

(defun callee (x &optional y) (list x y))
(defun caller (callee 23))
(defun callee (x) (list x))

Now, if we compiled caller with a direct call, it would turn out like (callee 23 nil).  But then, after the redefinition, we’d have to recompile caller.  Note this can go the other way as well — we could redefine callee to have more optional arguments, or even more fixed arguments (meaning that the call should now throw an exception).

Recompiling isn’t such a big deal, right?  The compiler is set up very naively: it just compiles every function that is invoked, and in this mode “recompilation” is equivalent to “just abandon the compiled code”.

Except… what do you do if caller is being run when callee is redefined?  Whoops!

Actually, of course, this is a known issue in JIT compilation, and one possible solution is “on-stack replacement” (“OSR”) — recompiling a function while it is running.

This to me seemed like a lot of bookkeeping, though: keeping a list of which functions to compile when some function was redefined, and figuring out a decent way to implement OSR.

The Plan

Instead I came up a with a simpler approach, involving — you guessed it — indirection.

On the callee side, I am going to keep the subr calling convention that is in place today.  This isn’t ideal in all cases, but it is reasonable for a lot of code.  Instead, all the changes will take place at spots where the JIT emits a call.

I am planning to have three kinds of function calls in the JIT:

  1. Indirect.  If we see some code where we can’t determine the callee, we’ll emit a call via Ffuncall like we do today.
  2. Fully direct.  There are some functions that are implemented in C, and that I think are unreasonable to redefine.  For these, we’ll just call the C function directly.  Another fully-direct case is where the code dispatches to a byte-code vector coming from the function’s constant pool — here, there’s no possibility to redefine the function, so we can simply always call the JIT-compiled form.
  3. Semi-direct.  This will be the convention used when JIT-compiled code calls via a symbol.

The core idea of a semi-direct call is to have multiple possible implementations of a function:

  • One “true” implementation.  If the function has 8 or fewer arguments (of any kind), it will simply have that many arguments.  The JIT will simply pretend that an optional argument is fixed.  If it has more than 8 arguments, following the subr convention it will just accept an array of arguments.
  • If the function has optional or rest arguments, there will be trampoline implementations with fewer arguments, that simply supply the required number of additional arguments and then call the true implementation.
  • Remember how there are exactly 10 relevant kinds of subr?  Any forms not covered by the above can simply throw an exception.

A vector of function pointers will be attached to each symbol, and so the JIT-compiled code can simply load the function pointer from the appropriate slot (a single load — the nice thing about a JIT is we can simply hard-code the correct address).

Then, when a function is redefined, we simply define any of the trampolines that are required as well.  We won’t even need to define all of them — only the ones that some actually-existing call site has needed.

Of course, no project like this is complete without a rathole, which is why instead of doing this I’m actually working on writing a compiler pre-pass so that the compiler itself can have the appropriate information about the callee at the point of call.  This sub-project turns out to feel a lot like writing a Java bytecode verifier…

Further Future

Currently the JIT is only used for lexically-bound bytecode functions.  That’s a reasonable restriction, I think — so one thing we should do is make sure that more of the Emacs core is using lexical binding.  Currently, only about 1/3 of the Lisp files in Emacs enable this feature; but many more easily could.

Once my current project is done, the JIT will have a decent calling convention by default.  Since we’ll have information about callees at points of call, I think it will be a good time to look into inlining.  This will require tackling recompilation (and perhaps OSR) and having some sort of tiered optimization approach.  There is still a lot for me to learn here — when does it make sense to inline?  And what metrics should I use to decide when some code is hot enough to optimize?  So, good times head even once the current project is done; and BTW if you have a reading list for any of this I would love to hear about it.

Once this is done, well, I have more ideas for even deeper JIT improvements.  Those will have to wait for another post.

Maintenance of an aging Bugzilla instance is a burden, and since CACAO development has mostly migrated to Bitbucket, the bug tracker will be maintained there as well.

The new location for tracking bugs is Bitbucket issues.

All the historical content from Bugzilla is statically archived at this site on the Bugzilla page.

I always used Eclipse extensively, although I moved away from it when it started having all sort of rendering issues with RHEL, mostly when SWT moved to GTK3 underneath. Most of those problems are slowly being fixed, and the IDE is again very usable under RHEL. I promised Lars last year that I would start using Eclipse again once those problem were addressed, and here I am, keeping the promise!

One thing I never really totally enjoyed in Eclipse was the debugger, though. Of all the IDEs I tried, I find the NetBeans debugger the absolute best, followed by IntelliJ IDEA. I don’t think the Eclipse debugger is bad in itself, but it doesn’t directly do the things I expect by default, and I often need to do quite a bit of tweaking in order to get it right. I admit I’m not exactly the average developer though, so it’s possible that what NetBeans offers me simply matches more what I find most useful. For instance, the detailed view for variables and the fact that you can quickly attach to the native portion of a process (and this works with the JDK as well as any hybrid application). Eclipse can do that, but requires some fiddling (most of the manual process I described on this post).

Now, there are things I love about Eclipse too. For example, while is true that it doesn’t show the most detailed view of variables, you can quickly execute arbitrary code right during the debugging sessions, simply by writing it. I’m not sure about IDEA, but NetBeans has similar functionality, but it’s hidden under tons of menu that you need to click and configure, and for very complex stuff the best option is extending the debugger itself, which is not trivial.

Recently, I’ve been debugging a weird issue in Fedora, and I found I was in need to scan a very large Map of fonts. The default formatter made things quite complex to understand while all I wanted was to quickly see what was the closest key in the map to the one I had as input.

You can see what I mean in this very trivial example.

Here my map is a simple immutable one with just 3 values, however already here you can see that the comma used to separate the values in the variable view makes things quite complicated, just imagine if this map had some thousands values, sorting them would have been quite an experience!

        Map<String, String> map =
                Map.of("Test1", "1,2,2",
                       "Test2", "2,3,1",
                       "Test3", "3,2,1");

Produces:

{Test3=3,2,1, Test1=1,2,2, Test2=2,3,1}

Not quite what I want!

The IDE gives us a very powerful tool though. Instead of simply changing the formatter, we can execute code to analyse the code! There is an hidden gem called “Display” view. You find it under “Window > Show View”, in the Debug perspective it should be readily visible, otherwise you can select “Other” and bring all the views up.

This neat view is simply a secondary editor that is active only during a debugging session, and allows you to instrument the code on the fly. What this means, is that you can type the following in the Display view and scan all the values for Map in question:

for (String key : map.keySet()) {
    System.err.println("* " + key + " -> " + map.get(key));
}

Then simply press “Execute selected text” and the code will execute. The subtle difference between “Execute” and “Display Result”, which is the icon next to the execute, is that the result of the code snippet will be printed in the Display, this is the result of the execution of the code itself, so in our case the snippet executes without a result, which is considered “false” by Eclipse apparently.

but they both have side effects on the running process, so for instance if you modify the map (well, that one is immutable, but you get the point) both options will modify it, no matter where the result gets printed. The Display doesn’t seem to like lambdas or default methods, so you need to keep things a bit old fashioned, but that’s hardly a problem.

Overall I find this feature extremely useful, and worth the hassle of dealing with the default configurations on the Eclipse Debugger. Just be careful though, side effects in debugging are hidden at every step, sometime even just a toString() called by the debugger can change your code in unexpected ways!

I attended the 5th Chrome Dev Summit this week. The talks were all recorded and are available via the schedule (the keynote and leadership panel on day 1 are perhaps of broadest interest and highest bang-for-buck viewing value). It was a high quality, well-produced event with an intimate feel – I was very surprised when Robert Nyman told me it was over 700 people! I appreciated the good vegetarian food options and noticed and was very impressed by the much-better-than-typical-tech-conferences gender representation and code of conduct visibility.

It doesn’t always look this way from the outside, but the various browser engine teams are more often than not working toward the same goals and in constant contact. For those who don’t know that, it was nice to see the shoutouts for other browsers and the use of Firefox’s new logo!

The focus of the event IMO was, as expected, the mobile Web. While the audience was Web developers, it was interesting to see what the Chrome team is focusing on. Some of the efforts felt like Firefox OS 4 years ago but I guess FxOS was just ahead of its time 😉

From my perspective, Firefox is in pretty good shape for supporting the things Chrome was promoting (Service Workers, Custom Elements and Shadow DOM, wasm, performance-related tooling, etc.). There are of course areas we can improve: further work on Fennec support for add-to-homescreen, devtools, and seeing a few things through to release (e.g. JS modules, Custom Elements, and Shadow DOM – work is underway and we’re hoping for soon!). Oh, and one notable exception to being aligned on things is the Network Information API that the Mozilla community isn’t super fond.

Other highlights for me personally included the Chrome User Experience Report (“a public dataset of key user experience metrics for popular origins on the web, as experienced by Chrome users under real-world conditions”) and the discussion about improving the developer experience for Web Workers.

It was great putting faces to names and enjoying sunny San Francisco (no, seriously, it was super sunny and hot). Thanks for the great show, Google!

It’s been a long road, but at last the puzzle is complete: Today we delivered Project Jigsaw for general use, as part of JDK 9.

Jigsaw enhances Java to support programming in the large by adding a module system to the Java SE Platform and to its reference implementation, the JDK. You can now leverage the key advantages of that system, namely strong encapsulation and reliable configuration, to climb out of JAR hell and better structure your code for reusability and long-term evolution.

Jigsaw also applies the module system to the Platform itself, and to the massive, monolithic JDK, to improve security, integrity, performance, and scalability. The last of these goals was originally intended to reduce download times and scale Java SE down to small devices, but it is today just as relevant to dense deployments in the cloud. The Java SE 9 API is divided into twenty-six standard modules; JDK 9 contains dozens more for the usual development and serviceability tools, service providers, and JDK-specific APIs. As a result you can now deliver a Java application together with a slimmed-down Java run-time system that contains just the modules that your application requires.

We made all these changes with a keen eye — as always — toward compatibility. The Java SE Platform and the JDK are now modular, but that doesn’t mean that you must convert your own code into modules in order to run on JDK 9 or a slimmed-down version of it. Existing class-path applications that use only standard Java SE 8 APIs will, for the most part, work without change.

Existing libraries and frameworks that depend upon internal implementation details of the JDK may require change, and they may cause warnings to be issued at run time until their maintainers fix them. Some popular libraries, frameworks, and tools — including Maven, Gradle, and Ant — were in this category but have already been fixed, so be sure to upgrade to the latest versions.

Looking ahead It’s been a long road to deliver Jigsaw, and I expect it will be a long road to its wide adoption — and that’s perfectly fine. Many developers will make use of the newly-modular nature of the platform long before they use the module system in their own code, and it will be easier to use the module system for new code rather than existing code.

Modularizing an existing software system can, in fact, be difficult. Sometimes it won’t be worth the effort. Jigsaw does, however, ease that effort by supporting both top-down and bottom-up migration to modules. You can thus begin to modularize your own applications long before their dependencies are modularized by their maintainers. If you maintain a library or framework then we encourage you to publish a modularized version of it as soon as possible, though not until all of its dependencies have been modularized.

Modularizing the Java SE Platform and the JDK was extremely difficult, but I’m confident it will prove to have been worth the effort: It lays a strong foundation for the future of Java. The modular nature of the platform makes it possible to remove obsolete modules and to deliver new yet non-final APIs in incubator modules for early testing. The improved integrity of the platform, enabled by the strong encapsulation of internal APIs, makes it easier to move Java forward faster by ensuring that libraries, frameworks, and applications do not depend upon rapidly-changing internal implementation details.

Learning more There are by now plenty of ways to learn about Jigsaw, from those of us who created it as well as those who helped out along the way.

If your time is limited, consider one or more of the following:

  • The State of the Module System is a concise, informal written overview of the module system. (It’s slightly out of date; I’ll update it soon.)

  • Make Way for Modules!, my keynote presentation at Devoxx Belgium 2015, packs a lot of high-level information into thirty minutes. I followed that up a year later with a quick live demo of Jigsaw’s key features.

  • Alex Buckley’s Modular Development with JDK 9, from Devoxx US 2017, covers the essentials in more depth, in just under an hour.

If you really want to dive in:

Comments, questions, and suggestions are welcome on the jigsaw-dev mailing list. (If you haven’t already subscribed to that list then please do so first, otherwise your message will be discarded as spam.)

Thanks! Project Jigsaw was an extended, exhilarating, and sometimes exhausting nine-year effort. I was incredibly fortunate to work with an amazing core team from pretty much the very beginning: Alan Bateman, Alex Buckley, Mandy Chung, Jonathan Gibbons, and Karen Kinnear. To all of you: My deepest thanks.

Key contributions later on came from Sundar Athijegannathan, Chris Hegarty, Lois Foltan, Magnus Ihse Bursie, Erik Joelsson, Jim Laskey, Jan Lahoda, Claes Redestad, Paul Sandoz, and Harold Seigel.

Jigsaw benefited immensely from critical comments and suggestions from many others including Jayaprakash Artanareeswaran, Paul Bakker, Martin Buchholz, Stephen Colebourne, Andrew Dinn, Christoph Engelbert, Rémi Forax, Brian Fox, Trisha Gee, Brian Goetz, Mike Hearn, Stephan Herrmann, Juergen Hoeller, Peter Levart, Sander Mak, Gunnar Morling, Simon Nash, Nicolai Parlog, Michael Rasmussen, John Rose, Uwe Schindler, Robert Scholte, Bill Shannon, Aleksey Shipilëv, Jochen Theodorou, Weijun Wang, Tom Watson, and Rafael Winterhalter.

To everyone who contributed, in ways large and small: Thank you!

Thanks to Alan Bateman and Alex Buckley
for comments on drafts of this entry.

For over twenty years the Java SE Platform and the JDK have evolved in large, irregular, and somewhat unpredictable steps. Each feature release has been driven by one or a few significant features, and so the schedule of each release has been adjusted as needed — sometimes more than once! — in order to accommodate the development of those features.

This approach made it possible to deliver big new features at a high level of quality, after thorough review and testing by early adopters. The downside, however, was that smaller API, language, and JVM features could only be delivered when the big features were ready.

This was an acceptable tradeoff in the decades before and after the turn of the century, when Java competed with just a few platforms which evolved at a similar stately pace. Nowadays, however, Java competes with many platforms which evolve at a more rapid pace.

For Java to remain competitive it must not just continue to move forward — it must move forward faster.

Back on the train Five years ago I mused in this space on the tension between developers, who prefer rapid innovation, and enterprises, which prefer stability, and the fact that everyone prefers regular and predictable releases.

To address these differing desires I suggested, back then, that we switch from the historical feature-driven release model to a time-driven “train” model, with a feature release every two years. In this type of model the development process is a continuous pipeline of innovation that’s only loosely coupled to the actual release process, which itself has a constant cadence. Any particular feature, large or small, is merged only when it’s nearly finished. If a feature misses the current train then that’s unfortunate but it’s not the end of the world, since the next train will already be waiting and will also leave on schedule.

The two-year train model was appealing in theory, but proved unworkable in practice. We took an additional eight months for Java 8 in order to address critical security issues and finish Project Lambda, which was preferable to delaying Lambda by two years. We initially planned Java 9 as a two-and-a-half year release in order to include Project Jigsaw, which was preferable to delaying Jigsaw by an additional eighteen months, yet in the end we wound up taking an additional year and so Java 9 will ship this month, three and a half years after Java 8.

A two-year release cadence is, in retrospect, simply too slow. To achieve a constant cadence we must ship feature releases at a more rapid rate. Deferring a feature from one release to the next should be a tactical decision with minor inconveniences rather than a strategic decision with major consequences.

So, let’s ship a feature release every six months.

That’s fast enough to minimize the pain of waiting for the next train, yet slow enough that we can still deliver each release at a high level of quality.

Proposal Taking inspiration from the release models used by other platforms and by various operating-system distributions, I propose that after Java 9 we adopt a strict, time-based model with a new feature release every six months, update releases every quarter, and a long-term support release every three years.

  • Feature releases can contain any type of feature, including not just new and improved APIs but also language and JVM features. New features will be merged only when they’re nearly finished, so that the release currently in development is feature-complete at all times. Feature releases will ship in March and September of each year, starting in March of 2018.

  • Update releases will be strictly limited to fixes of security issues, regressions, and bugs in newer features. Each feature release will receive two updates before the next feature release. Update releases will ship quarterly in January, April, July, and October, as they do today.

  • Every three years, starting in September of 2018, the feature release will be a long-term support release. Updates for these releases will be available for at least three years and quite possibly longer, depending upon your vendor.

In this model the overall rate of change should be about the same as it is today; what’s different is that there will be many more opportunities to deliver innovation. The six-month feature releases will be smaller than the multi-year feature releases of the past, and therefore easier to adopt. Six-month feature releases will also reduce the pressure to backport new features to older releases, since the next feature release will never be more than six months away.

Developers who prefer rapid innovation, so that they can leverage new features in production as soon as possible, can use the most recent feature release or an update release thereof and move on to the next one when it ships. They can deliver an application in a Docker image, or other type of container package, along with the exact Java release on which the application will run. Since the application and the Java release can always be tested together, in a modern continuous-integration and continuous-deployment pipeline, it will be straightforward to move from one Java release to the next.

Enterprises that prefer stability, so that they can run multiple large applications on a single shared Java release, can instead use the current long-term support release. They can plan ahead to migrate from one long-term support release to the next, like clockwork, every three years.

To make it clear that these are time-based releases, and to make it easy to figure out the release date of any particular release, the version strings of feature releases will be of the form $YEAR.$MONTH. Thus next year’s March release will be 18.3, and the September long-term support release will be 18.9.

Implications This proposal will, if adopted, require major changes in how contributors in the OpenJDK Community produce the JDK itself; I’ve posted some initial thoughts as to how we might proceed there. It will be made easier if we can reduce the overhead of the Java Community Process, which governs the evolution of the Java SE Platform; my colleagues Brian Goetz and Georges Saab have already raised this topic with the JCP Executive Committee.

This proposal will, ultimately, affect every developer, user, and enterprise that relies upon Java. It will, if successful, help Java remain competitive — while maintaining its core values of compatibility, reliability, and thoughtful evolution — for many years to come.

Comments and questions are welcome, either on the OpenJDK general discussion list (please subscribe to that list in order to post to it) or on Twitter, with the hashtag #javatrain.

After almost fifteen years I have decided to quit working on IKVM.NET. The decision has been a long time coming. Those of you that saw yesterday’s Twitter spat, please don’t assume that was the cause. It rather shared an underlying cause. I’ve slowly been losing faith in .NET. Looking back, I guess this process started with the release of .NET 3.5. On the Java side things don’t look much better. The Java 9 module system reminds me too much of the generics erasure debacle.

I hope someone will fork IKVM.NET and continue working on it. Although, I’d appreciate it if they’d pick another name. I’ve gotten so much criticism for the name over the years, that I’d like to hang on to it 😊

I’d like to thank the following people for helping me make this journey or making the journey so much fun: Brian Goetz, Chris Brumme, Chris Laffra, Dawid Weiss, Erik Meijer, Jb Evain, John Rose, Mads Torgersen, Mark Reinhold, Volker Berlin, Wayne Kovsky, The GNU Classpath Community, The Mono Community.

And I want to especially thank my friend Miguel de Icaza for his guidance, support, inspiration and tireless efforts to promote IKVM.

Thank you all and goodbye.

[NOTE: This article talks about commercial products and contains links to them, I do not receive any money if you buy those tools, nor I work for or I am affiliated to any of those companies. The opinion expressed here are mine and the review is subjective]

This is my attempt at a review of Spitfire Audio BT Phobos. Before diving into the review, and since I know I will be critic particularly on some aspects, I think it’s fair to assess the plugin right away: BT Phobos is an awesome tool, make no mistakes.

BT Phobos is a “polyconvolution” synthesiser. It is, in fact, the first “standalone” plugin produced by Spitfire Audio, which is one of the companies I respect the most when it comes to music production and sample based instruments.

The term polyconvolution is used by the Spitfire Audio team to indicate the simultaneous use of three convolvers for four primary audio paths: you can send any amount of each of those four primary sources (numbered 1 to 4) outputs to each of the three convolution engines (named W, X and Y).

Screen Shot 2017-04-18 at 13.31.39

Source material controls

There is lot of flexibility in the mixing capabilities; there are, of course, separate dry/wet signal knobs that send a specific portion of the unprocessed source material to the “amplifier” module, control how much of the signal goes to the convolution circuits, and finally how much of each of the convolution engines applies to each of the source sound.

This last bit is achieved by means of an interesting nabla shaped X/Y pad: by positioning the icon that represents the source module closer to a corner it’s possible activate just the convolution engine that represents that corner; for example, top left is the W engine, top right the X and bottom the Y. Manually moving the icon gradually introduces contributions from the other engines, and double clicking on the icon makes all convolvers contribute equally to the wet sound, by positioning them to the center of the nabla.

Screen Shot 2017-04-18 at 15.45.15

The convolution mixer

Finally, each convolver has a control that allows to change the output level of the convolution engine before it reaches its envelope shaper. Spitfire Audio has released a very interesting flow diagram that shows the signal path in detail, which is linked below for reference.

BT Phobos signal path

In addition to the controls just described, the main GUI has basic controls to tweak the source material with an ADSR envelope which is directly accessible below each of the main sound sources as well as the convolutions modules, but it’s possible to have access to more advanced settings by clicking on the number or the letter that identifies the module name.

Screen Shot 2017-04-18 at 13.32.21

The advanced controls interface

An example of such controls is the Hold parameter, which let the user adjust the time the sound is held at full level before entering the Decay phase of its envelope; another useful tool is the sampling and IR offset controls, which allow to tweak parameters like the starting point of the material or the quantisation and its Speed (the playback speed for the samples, and is a function of the host tempo); there is also a control to influence the general pitch of the sound; finally a simple but effective section is dedicated to filtering – although a proper EQ is missing – as well as panning and level adjustments.

All those parameters are particularly important settings when using loops, but also contribute to shaping the sound with the pitched material, and can be randomised for interesting effects and artefacts generated from the entropy (you can just randomise the material selection only as opposed to all the parameters).

Modulation is also present, of course, with various LFOs of various kind that can be used to modulate basically everything. You can access them either by clicking on the mappings toggle below the ADSR envelope of each section, or by using the advanced settings pages.

The amount of tweaks that can be made to the material in both the source and the convolution engines is probably the most important aspect of BT Phobos, since it gives an excellent amount of freedom to create new sounds from what’s available, which is already a massive amount of content, and allows to build wildly different patches with a bit of work, but it’s definitely not straightforward and needs time to understand the combined effects that each setting has on the whole.

Since the material is polyphonic, the Impulse Responses for the convolution are created on the fly, and in fact, one interesting characteristic of BT Phobos is that there is no difference between a material for the convolution engines and one for the source module,  both draw from the same pool of sounds.

Screen Shot 2017-04-18 at 14.41.57

BT Phobos beautiful GUI

There is a difference on the type of material though, where loop based samples are, well, looped (and tempo sync’ed), and their pitch does not change based on the key that triggers them (although you can still affect the general pitch of the sound with the advanced controls), “tonal” material are pitched and change following the midi notes.

One note about the LFOs: the mappings are “per module”. In other words, it is possible to modulate almost every parameter inside a single module, be it one of the four input sources or one of the three convolution engines, but there seem to be no way to define a global mapping of some kind. For example, I found a very nice patch from Mr. Christian Henson (which incidentally made, at least in my opinion, the best and most balanced overall presets), and I noticed I could make it even more interesting by using the modulation wheel. I wanted to modulate the CC1 message with an LFO (in fact, ideally it would be even better to have access to a custom envelope, but BT Phobos doesn’t have any for modulation use), but I could not find a way to do that other than using Logic’s own Midi FX. I understand that MIDI signals are generated outside the scope of the plugin, but it would be fantastic to have the option of tweaking and modulate everything from within the synth itself.

All the sources and convolvers can be assigned to separate parts of the keyboard by tweaking the mapper at the bottom of the GUI. It is not possible to map a sound to start from an offset in the keyboard controls – for example to play C1 on the keyboard but trigger C2, or any other note – but of course you can change the global pitch so this has effectively the same result, and as said before it can also be modulated with an LFO or via DAW automation, for more interesting effects.

Screen Shot 2017-04-18 at 21.42.19

Keyboard mapping tool

Indeed, the flexibility of the tool, and the number of options at disposal for tweaking the sounds are very impressive. Most patches are very nice and ready to be used as they are, and blend nicely with lots of disparate styles. Some patches are very specific though, and pose a challenge to be used. Generally, I would consider these as starting points for exploration, rather than “final”.

When reading about BT Phobos in the weeks before its release many people asked whether you could add your own sound to it or not. It’s not possible, unfortunately.

At first, I thought that wasn’t a limitation or a deal breaker. I still think it’s not a deal breaker, but I see the value added that BT Phobos has even just as a standalone synth, as opposed to recreate the same kind of signal path manually with external tools, to give your own content the “Phobos treatment”, which is something that is entirely possible of course, for example just with Alchemy and Space Designer (which are both included in MainStage, so you can get them for a staggering 30 euros if you are a Mac user, even if you don’t use Logic Pro X!), but of course, we would be trading away the immediacy that BT Phobos delivers.

That, maybe, is my main criticism to this synth, and I hope Spitfire Audio turns BT Phobos into a fully fledged tool for sound design over time, maybe enabling access to spectral shaping in some form or another, so we can literally paint over (or paint away!) portions of the sound, which is something you can do with iZotope Iris or Alchemy and is a very powerful way to shape a sound and do sound design in general.

Another thing that is missing is a sound effect module, although I don’t know how important that is, given that there are thousands of outstanding plugins that do all sort of effects from delay to chorus etc… And, in fact, many patches benefit for added reverb (I use Eventide Blackhole and found that works extremely well with BT Phobos, since it’s also prominently used for weird sound effects). But it may be interesting to play by putting some effects (including a more proper EQ section) in various places in the signal path, although it’s all too easy to generate total chaos from such experimentation, so it’s possible the Spitfire Audio simply thought to leave this option for another time and instead focus on a better overall experience.

And there’s no arpeggiator! Really!

The number of polyphonic voices can be altered. Spitfire Audio states that the synth tweaks the number of voices at startup to match the characteristics of your computer, but I can’t confirm that, since every change I do seems to remain, even if I occasionally hear some pop and cracks at higher settings. Nevertheless, the CPU usage is pretty decent unless you go absolutely crazy with the polyphony count. I also noted that the numbers effect the clarity of the sound. This is understandable since an higher count means more notes can be generated at the same time, which means more things are competing for the same spectrum, and things can become very confusing very quickly. On the other end, a lower polyphony count has a bad impact on how the notes are generated. I feel sometime that things just stop generating sound, which is counter intuitive and very disturbing, especially since it’s very easy to have a high polyphony count with all those sources and convolvers.

Also to note is that, by nature, some patches have very wild difference in their envelopes and level settings, which means it’s all to easy to move from a quiet to a very loud patch just by clicking “next” (which is possible in Logic at least with the next/prev patch buttons on top of the plugin main frame). The synth does not stop the sound, nor does any attempt to fade from one sound to the next, instead, the convolutions simply keep working on the next sample in queue with the new settings! I still have to decide if this is cool or not, perhaps it’s not intentional, but I can see how this could be used to automate patch changes in some clever way during playback! And indeed, a was able to create a couple of interesting side effects just by changing between patches at the right time.

More on the sounds. The amount of content is really staggering, and simply cycling through the patches does not make justice to this synth, at all!

What BT Phobos wants is a user that spends time tweaking the patches and play with the source material to get the most out it, however it’s easy to see how limiting this may feel at the same time, particularly with the more esoteric and atonal sounds, and there’s certainly a limit on how good a wood stick convolved with an aluminium thin can may sound, so indeed some patches do feel repetitive at times, as the source material does. There are quite a few very similar drum loops for example, or various pitches “wind blowing into a pipe” kind of things.

This is a problem common to other synths based on the idea of tweaking sounds from the environment, though. For example, I have the amazing Geosonics from Soniccouture, which is an almost unusable library that, once tweaked, is capable of amazing awesomeness. Clearly, the authors of both synths – but this is especially valid for BT Phobos I think – are looking at an audience that is capable of listening through the detuned and dissonant sound waves and shape a new form of music.

This is probably the reason why so many of the pre assembled patches dive the user full speed into total sound design territory; however, and this is another important point of criticism, this is sound design that has already been done for you… A lot of the BT patches, in particular, are clearly BT patches, using them as they are means you are simply redoing something that has already been done before, and, despite with a very experimental feeling still strongly present, it’s not totally unheard or new.

For example, I also happen to have Break Tweaker and Stutter Edit (tools that also originally come from BT), and I could not resist to the temptation to play something that resembles BT work on “This Binary Universe” or “_” (fantastic albums)! While this seems exciting – BT in a box! And you can also see the democratising aspect of BT Phobos, I can do that in half hour instead of six months of manual CSound programming! – it’s an unfortunate and artificial limitation on a tool that is otherwise a very powerful enabler, capable of bringing complex sound design one step closer to the general public. Having the ability to process your own sounds would mitigate this aspect I think.

I do see how this is useful for a composer in need of a quick solution for an approaching deadline even with the most experimental tones, though: those patches can resolve a deadlock or take you out of an impasse in a second.

The potential for BT Phobos to become a must have tool for sound design are all there, especially if Spitfire Audio keeps adding content, perhaps more varied (and even better, allow to load your own content). The ability to shape the existing sounds already make it very usable. I don’t think it’s a general tool at this stage, though, and definitely it should not be the first synth or sound shaping processor in your arsenal, especially if you are starting out now.

But it’s not just a one trick pony either, it does offer you quite a lot of possibilities, and the more you work on that, the more addictive it becomes, and I can see Spitfire Audio offering soon this synth within a collection comprising of some of their more experimental stuff like LCO and Enigma, which would be very nice, indeed.

It’s unfortunate that Spitfire Audio does not offer an evaluation period: contrary to most of their offering, BT Phobos needs time to be fully grasped and it’s all but immediate (well, unless you are happy with the default patches or you really just need to “get out of troubles” quickly, but be careful with that because the tax is on the originality), but it can, and does, evolve, as its convolutions do, over time and it can absolutely deliver total awesomeness if used correctly.

Most patches are also usable out of the box, and especially by adding some reverb or doing some post processing with other tools, it’s possible to squeeze even more life out of them.

Overall, I do recommend BT Phobos, is a wonderful, very addictive synthesiser.

Quantum Curling

Last week we had a work week at Mozilla’s Toronto office for a bunch of different projects including Quantum DOM, Quantum Flow (performance), etc. It was great to have people from a variety of teams participate in discussions and solidify (and change!) plans for upcoming Firefox releases. There were lots of sessions going on in parallel and I wasn’t able to attend them all but some of the results were written up by the inimitable Ehsan in his fourth Quantum Flow newsletter.

Near the end of the week, Ehsan gave an impromptu walkthrough of the Gecko profiler. I’m planning on taking some of the tips he gave and that were discussed and put them onto the documentation for the profiler. If you’re interested in helping, please let me know!

The photo above is of us going curling at the High Park Curling Club. It was a lot of fun and I was happy that only one other person had ever curled before so it was a unique experience for almost everyone!

As previously reported, the JSR 269 annotation processing APIs in the javax.lang.model and javax.annotation.processing packages are undergoing maintenance review as part of Java SE 9.

All the planned changes to the JSR 269 API are in JDK 9 build 164, downloadable as an early access binary. Of note new in build 164 is the annotation type javax.annotation.processing.Generated, meant to be a drop-in replacement for javax.annotation.Generated since the latter is not in a convenient module.

Please try out your existing annotation processors -- compiling them, running them, etc. -- on JDK 9 and report your experiences, good or bad, to compiler-dev@openjdk.java.net.

As has been done previously during Java SE 7 and Java SE 8, the JSR 269 annotation processing API is undergoing a maintenance review (MR) as part of Java SE 9.

Most of the API changes are in support of adding modules to the platform, both as a language structure in javax.lang.model.* as well as another interaction point in javax.annotation.processing in the Filer and elsewhere. A small API change was also done to better support repeating annotations. A more detailed summary of the API changes is included in the MR material.

The API changes are intended to be largely compatible with the sources of existing processors, their binary linkage, as well as their runtime behavior. However, it would be helpful to verify that your existing processors work as expected when run under JDK 9. JDK 9 early access binaries are available for download. Please report experiences running processors under JDK 9 as comments here or to me as email. Feedback on the API changes can be sent to compiler-dev@openjdk.java.net.

After turning off comments on this blog a few years ago, the time has now come to remove all the posts containing links. The reason is again pretty much the same as it was when I decided to turn off the comments - I still live in Hamburg, Germany.

So, I've chosen to simply remove all the posts containing links. Unfortunately, that were pretty much all of them. I only left my old post up explaining why this blog allows no comments, now updated to remove all links, of course.

Over the past years, writing new blog posts here has become increasingly rare for me. Most of my 'social media activity' has long moved over to Twitter.

Unfortunately, I mostly use Twitter as a social bookmarking tool, saving and sharing links to things that I find interesting.

As a consequence, I've signed up for a service that automatically deletes my tweets after a short period of time. I'd link to it, but ...

If you’ve been following Infinity and would like to, you know, download some code and try it out… well, now you can!

screenshot

The first release candidate is finally available. It can be downloaded here or from NuGet.

What's New (relative to IKVM.NET 8.0):

  • Integrated OpenJDK 8u45.
  • Many fixes to late binding support.
  • Added ikvmc support for deterministic output files.
  • Various sun.misc.Unsafe improvements.
  • Many minor bug fixes and performance tweaks.

Changes since previous development snapshot:

  • Assemblies are strong named.
  • Fix for bug #303. ikvmc internal compiler error when trying to get interfaces from type from missing assembly reference.
  • Implemented NIO atomic file move on Windows.

Binaries available here: ikvmbin-8.1.5717.0.zip

Sources: ikvmsrc-8.1.5717.0.zip, openjdk-8u45-b14-stripped.zip

Thanks to everybody who commented on the JamVM 2.0.0 release, and apologies it's taken so long to approve them - I was expecting to get an email when I had an unmoderated comment but I haven't received any.

To answer the query regarding Nashorn.  Yes, JamVM 2.0.0 can run Nashorn.  It was one of the things I tested the JSR 292 implementation against.  However, I can't say I ran any particularly large scripts with it (it's not something I have a lot of experience with).  I'd be pleased to hear any experiences (good or bad) you have.

So now 2.0.0 is out of the way I hope to do much more frequent releases.  I've just started to look at OpenJDK 9.  I was slightly dismayed to discover it wouldn't even start up (java -version), but it turned out to be not a lot of work to fix (2 evenings).  Next is the jtreg tests...

I'm pleased to announce a new release of JamVM.  JamVM 2.0.0 is the first release of JamVM with support for OpenJDK (in addition to GNU Classpath). Although IcedTea already includes JamVM with OpenJDK support, this has been based on periodic snapshots of the development tree.

JamVM 2.0.0 supports OpenJDK 6, 7 and 8 (the latest). With OpenJDK 7 and 8 this includes full support for JSR 292 (invokedynamic). JamVM 2.0.0 with OpenJDK 8 also includes full support for Lambda expressions (JSR 335), type annotations (JSR 308) and method parameter reflection.

In addition to OpenJDK support, JamVM 2.0.0 also includes many bug-fixes, performance improvements and improved compatibility (from running the OpenJDK jtreg tests).

The full release notes can be found here (changes are categorised into those affecting OpenJDK, GNU Classpath and both), and the release package can be downloaded from the file area.

JNode, the free operating system developed in Java, has now its twitter account. Follow #JNode !

GitHub is now the main repository for JNode sources : https://github.com/jnode

. We are now using GitHub’s issue tracker.