Planet Classpath
     ____               
    /    \              
   |-. .-.|             
   (_@)(_@)             
   .---_  \             
  /..   \_/             
  |__.-^ /              
      }  |              
     |   [              
     [  ]               
    ]   |               
    |   [               
    [  ]                
   /   |        __      
  \|   |/     _/ /_     
 \ |   |//___/__/__/_   
\\  \ /  //    -____/_  
//   "   \\      \___.- 
 //     \\  __.----._/_ 
/ //|||\\ .-         __>
[        /         __.- 
[        [           }  
\        \          /   
 "-._____ \.____.--"    
    |  | |  |           
    |  | |  |           
    |  | |  |           
    |  | |  |           
    {  } {  }           
    |  | |  |           
    |  | |  |           
    |  | |  |           
    /  { |  |           
 .-"   / [   -._        
/___/ /   \ \___"-.     
    -"     "-           

strace patch.

Important Graphos fix.

Graphos had issues when printing and the view was not 100%: to speed up drawRect, all objects were represented scaled, so that they had not to be scaled each time, which especially for Bezier Paths is expensive with all the associated handles.

Thie issue is finally fixed by either caching both original and zoomed values for each object and conditionally drawing them depending on the drawingContext.

Here the proof with GSPdf showing the generated PDF!



Soon a new release then!

I’ve been working a bit more on my Emacs JIT, in particular on improving function calling.  This has been a fun project so I thought I’d talk about it a bit.

Background

Under the hood, the Emacs Lisp implementation has a few different ways to call functions.  Calls to or from Lisp are dispatched depending on what is being called:

  • For an interpreted function, the arguments are bound and then the interpreter is called;
  • For a byte-compiled function using dynamic binding, the arguments are bound and then the bytecode interpreter is called;
  • For a byte-compiled function using lexical binding, an array of arguments is passed to the bytecode interpreter;
  • For a function implemented in C (called a “subr” internally), up to 8 arguments are supported directly — as in, C functions of the form f(arg,arg,...); for more than that, an array of arguments is passed and the function itself must decide which slot means what.  That is, there are exactly 10 forms of subr (actually there are 11 but the one missing from this description is used for special forms, which we don’t need to think about here).

Oh, let’s just show the definition so you can read for yourself:

union {
Lisp_Object (*a0) (void);
Lisp_Object (*a1) (Lisp_Object);
Lisp_Object (*a2) (Lisp_Object, Lisp_Object);
Lisp_Object (*a3) (Lisp_Object, Lisp_Object, Lisp_Object);
Lisp_Object (*a4) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
Lisp_Object (*a5) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
Lisp_Object (*a6) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
Lisp_Object (*a7) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
Lisp_Object (*a8) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
Lisp_Object (*aUNEVALLED) (Lisp_Object args);
Lisp_Object (*aMANY) (ptrdiff_t, Lisp_Object *);
} function;

Initial Approach

Initially the JIT worked like a lexically-bound bytecode function: an array of arguments was passed to the JIT-compiled function.  The JIT compiler emitted a bunch of code to decode the arguments.

For Lisp functions taking a fixed number of arguments, this wasn’t too bad — just moving values from fixed slots in the array to fixed values in the IR.

Handling optional arguments was a bit uglier, involving a series of checks and branches, so that un-bound arguments could correctly be set to nil.  These were done something like:

if nargs < 1 goto nope1
arg0 = array[0]
if nargs < 2 goto nope2
arg1 = array[1]
goto first_bytecode
nope1: arg0 = nil
nope2: arg1 = nil
first_bytecode: ...

&rest arguments were even a bit worse, requiring a call to create a list.  (This, I think, can’t be avoided without a much smarter compiler, one that would notice when reifying the list could be avoided.)

Note that calling also has to use the fully generic approach: we make a temporary array of arguments, then call a C function (Ffuncall) that does the dispatching to the callee.  This is also a source of inefficiency.

Today

Recently, I changed the JIT from this approach to use the equivalent of the subr calling convention.  Now, any function with 8 or fewer (non-&rest) arguments is simply an ordinary function of N arguments, and we let the already-existing C code deal with optional arguments.

Although this often makes the generated assembly simpler, it won’t actually perform any better — the same work is still being done, just somewhere else.  However, this approach does use a bit less memory (most JIT-compiled functions are shorter); and it opens the door to an even bigger improvement.

The Future

What I’m implementing now is an approach to removing most of the overhead from JIT-compiled function calls.

Now, ideally what I’d like is to have every call site work “like C”: move the arguments to exactly where the callee expects them to be, and then call.  However, while looking at this I found some problems that make it tricky:

  • We still need to be able to call Lisp functions from C, so we’re limited to, at best, subr-style calling conventions;
  • While &rest arguments are straightforward (in our simple compiler, somebody just has to make the list); &optional arguments don’t have a good C-like analog.  The callee could push extra arguments, but…
  • In Lisp, a function can be redefined at any time, and it is fine to change the function’s signature.

Consider this example:

(defun callee (x &optional y) (list x y))
(defun caller (callee 23))
(defun callee (x) (list x))

Now, if we compiled caller with a direct call, it would turn out like (callee 23 nil).  But then, after the redefinition, we’d have to recompile caller.  Note this can go the other way as well — we could redefine callee to have more optional arguments, or even more fixed arguments (meaning that the call should now throw an exception).

Recompiling isn’t such a big deal, right?  The compiler is set up very naively: it just compiles every function that is invoked, and in this mode “recompilation” is equivalent to “just abandon the compiled code”.

Except… what do you do if caller is being run when callee is redefined?  Whoops!

Actually, of course, this is a known issue in JIT compilation, and one possible solution is “on-stack replacement” (“OSR”) — recompiling a function while it is running.

This to me seemed like a lot of bookkeeping, though: keeping a list of which functions to compile when some function was redefined, and figuring out a decent way to implement OSR.

The Plan

Instead I came up a with a simpler approach, involving — you guessed it — indirection.

On the callee side, I am going to keep the subr calling convention that is in place today.  This isn’t ideal in all cases, but it is reasonable for a lot of code.  Instead, all the changes will take place at spots where the JIT emits a call.

I am planning to have three kinds of function calls in the JIT:

  1. Indirect.  If we see some code where we can’t determine the callee, we’ll emit a call via Ffuncall like we do today.
  2. Fully direct.  There are some functions that are implemented in C, and that I think are unreasonable to redefine.  For these, we’ll just call the C function directly.  Another fully-direct case is where the code dispatches to a byte-code vector coming from the function’s constant pool — here, there’s no possibility to redefine the function, so we can simply always call the JIT-compiled form.
  3. Semi-direct.  This will be the convention used when JIT-compiled code calls via a symbol.

The core idea of a semi-direct call is to have multiple possible implementations of a function:

  • One “true” implementation.  If the function has 8 or fewer arguments (of any kind), it will simply have that many arguments.  The JIT will simply pretend that an optional argument is fixed.  If it has more than 8 arguments, following the subr convention it will just accept an array of arguments.
  • If the function has optional or rest arguments, there will be trampoline implementations with fewer arguments, that simply supply the required number of additional arguments and then call the true implementation.
  • Remember how there are exactly 10 relevant kinds of subr?  Any forms not covered by the above can simply throw an exception.

A vector of function pointers will be attached to each symbol, and so the JIT-compiled code can simply load the function pointer from the appropriate slot (a single load — the nice thing about a JIT is we can simply hard-code the correct address).

Then, when a function is redefined, we simply define any of the trampolines that are required as well.  We won’t even need to define all of them — only the ones that some actually-existing call site has needed.

Of course, no project like this is complete without a rathole, which is why instead of doing this I’m actually working on writing a compiler pre-pass so that the compiler itself can have the appropriate information about the callee at the point of call.  This sub-project turns out to feel a lot like writing a Java bytecode verifier…

Further Future

Currently the JIT is only used for lexically-bound bytecode functions.  That’s a reasonable restriction, I think — so one thing we should do is make sure that more of the Emacs core is using lexical binding.  Currently, only about 1/3 of the Lisp files in Emacs enable this feature; but many more easily could.

Once my current project is done, the JIT will have a decent calling convention by default.  Since we’ll have information about callees at points of call, I think it will be a good time to look into inlining.  This will require tackling recompilation (and perhaps OSR) and having some sort of tiered optimization approach.  There is still a lot for me to learn here — when does it make sense to inline?  And what metrics should I use to decide when some code is hot enough to optimize?  So, good times head even once the current project is done; and BTW if you have a reading list for any of this I would love to hear about it.

Once this is done, well, I have more ideas for even deeper JIT improvements.  Those will have to wait for another post.

We are pleased to announce the release of IcedTea 3.7.0!

The IcedTea project provides a harness to build the source code from OpenJDK using Free Software build tools, along with additional features such as the ability to build against system libraries and support for alternative virtual machines and architectures beyond those supported by OpenJDK.

This release updates our OpenJDK 8 support with the January 2018 security fixes from OpenJDK 8 u161.

If you find an issue with the release, please report it to our bug database under the appropriate component. Development discussion takes place on the distro-pkg-dev OpenJDK mailing list and patches are always welcome.

Full details of the release can be found below.

What’s New?

New in release 3.7.0 (2018-02-28)

  • Security fixes
  • Import of OpenJDK 8 u152 build 16
    • S4953367: MAWT: Java should be more careful manipulating NLSPATH, XFILESEARCHPATH env variables
    • S6772009: java/util/concurrent/locks/ReentrantLock/CancelledLockLoops.java test failed with ‘Completed != 2′
    • S6947916: JarURLConnection does not handle useCaches correctly
    • S6980209: Make tracking SecondaryLoop.enter/exit methods easier
    • S7052625: com/sun/net/httpserver/bugs/6725892/Test.java fails intermittently
    • S7102702: java/net/PortUnreachableException/OneExceptionOnly.java failing
    • S7131356: (props) “No Java runtime present, requesting install” when creating VM from JNI [macosx]
    • S7172652: With JDK 1.7 text field does not obtain focus when using mnemonic Alt/Key combin
    • S8022291: Mac OS: Unexpected JavaLaunchHelper message displaying
    • S8022580: sun.net.ftp.impl.FtpClient.nameList(String path) handles “null” incorrectly
    • S8025306: Inet[4|6]Address class and fieldID initialization in networking native code
    • S8028363: XmlGregorianCalendarImpl.getTimeZone() bug when offset is less than 10 minutes
    • S8028725: [Parfait] warnings from b116 for jdk.src.solaris.native.java.net: JNI pending exceptions
    • S8028769: Check src/share/native/java/lang/*.c for cases when JNI calls are used with pending warnings
    • S8029018: (bf) Check src/share/native/java/nio/Bits.c for JNI pending exceptions
    • S8030116: (process) Check Parfait warnings in src/solaris/native/java/lang/*
    • S8031586: Check jdk/src/*/native/com/sun/security/auth/module for pending JNI exceptions
    • S8033371: jdk/src/solaris/native/java/lang/ProcessEnvironment.c environ doesn’t check for JNI pending exceptions
    • S8033372: Check jdk/src/share/native/java/lang for JNI pending exception issues
    • S8034102: Check solaris/native/sun/nio/fs/MacOSXNativeDispatcher.c for JNI pending exceptions
    • S8034249: need more workarounds for suspend equivalent condition issue
    • S8035158: Remove dependency on sun.misc.RegexpPool and friends
    • S8035568: [macosx] Cursor management unification
    • S8035653: InetAddress.getLocalHost crash
    • S8037346: Need to terminate server process if client runs into problems
    • S8037909: JNI warnings in jdk/src/windows/native/java/nio/MappedByteBuffer.c
    • S8038075: JNI warnings in jdk/src/share/native/sun/misc/VMSupport.c
    • S8038348: Instance field load is replaced by wrong data Phi
    • S8039412: Stack overflow on Linux using DialogTypeSelection.NATIVE
    • S8040322: TextArea.replaceRange() and insert() are broken with setText(null)
    • S8040635: [macosx] Printing a shape filled with a texture doesn’t work under Mac OS X
    • S8043913: remove legacy code in SPARC’s VM_Version::platform_features
    • S8046768: com/sun/jndi/ldap/LdapTimeoutTest.java fails intermittently
    • S8046893: JNI exception pending in jdk/src/solaris/native/java/net: ExtendedOptionsImpl.c, PlainDatagramSocketImpl.c
    • S8046901: Check jdk/src/solaris/native/sun/nio for Parfait flagged uninitialized memory
    • S8049717: expose L1_data_cache_line_size for diagnostic/sanity checks
    • S8050478: [macosx] Cursor not updating correctly after closing a modal dialog
    • S8054214: JapaneseEra.getDisplayName doesn’t return names if it’s an additional era
    • S8055204: Memory leak in jdk/src/windows/native/java/lang/java_props_md.c
    • S8058316: lookupDefaultPrintService returns null on Solaris 11
    • S8061258: [macosx] PrinterJob’s native Print Dialog does not reflect specified Copies or Page Ranges
    • S8064470: JNI exception pending in jdk/src/java/base/unix/native/libjava/FileDescriptor_md.c
    • S8066250: compiler/dependencies/MonomorphicObjectCall/TestMonomorphicObjectCall.java fails product
    • S8067059: PrinterJob.pageDialog() with DialogSelectionType.NATIVE returns a PageFormat when cancelled.
    • S8071424: JCK test api/java_net/Socket/descriptions.html#Bind crashes on Windows
    • S8075516: Deleting a file from either the open or save java.awt.FileDialog hangs.
    • S8076249: NPE in AccessBridge while editing JList model
    • S8076554: [macosx] Custom Swing text components need to allow standard accessibility
    • S8080486: JNI exception pending in jdk/src/java.base/windows/native/libnet/DualStackPlainSocketImpl.c
    • S8081219: hs_err improvement: Add event logging for class redefinition to the hs_err file
    • S8087144: sun/security/krb5/auto/MaxRetries.java fails with Retry count is -1 less
    • S8087342: Crash in klassItable::initialize_itable_for_interface
    • S8089573: [macosx] Incorrect char to glyph mapping printing on OSX 10.10
    • S8129361: ISO 4217 amendment 160
    • S8130041: TsacertOptionTest.java intermittently fails on Mac
    • S8132209: DiagnosticCommandImpl.getNotificationInfo() may expose internal representation
    • S8132455: com/sun/jndi/ldap/LdapTimeoutTest.java fails at handleNamingException
    • S8134119: Use new API to get cache line sizes
    • S8134389: Crash in HotSpot with jvm.dll+0x42b48 ciObjectFactory::create_new_metadata
    • S8134918: C2: Type speculation produces mismatched unsafe accesses
    • S8135124: com/sun/jndi/ldap/LdapTimeoutTest.java failed intermittently
    • S8136570: Stop changing user environment variables related to /usr/dt
    • S8139189: VK_OEM_102 dead key detected as VK_UNDEFINED
    • S8140309: [REDO] failed: no mismatched stores, except on raw memory: StoreB StoreI
    • S8140525: AwtFrame::WmShowWindow() may steal focus
    • S8141370: com/sun/jndi/ldap/LdapTimeoutTest.java failed intermittently
    • S8143897: Weblogic12medrec assert(handler_address == SharedRuntime::compute_compiled_exc_handler(nm, pc, exception, force_unwind, true)) failed: Must be the same
    • S8145207: [macosx] JList, VO can’t access non-visible list items
    • S8145732: Duplicate entry in http.nonProxyHosts will ignore subsequent entries
    • S8145952: ISO 4217 amendment 161
    • S8145981: (fs) LinuxWatchService can reports events against wrong directory
    • S8146961: Fix PermGen memory leaks caused by static final Exceptions
    • S8147772: Update KerberosTicket to describe behavior if it has been destroyed and fix NullPointerExceptions
    • S8149521: automatic discovery of LDAP servers with Kerberos authentication
    • S8149714: [parfait] char array lengths don’t match in awt_Font.cpp:1701
    • S8152000: Java FlightRecorder does not run on Java 8 compact 3 profile since u65
    • S8152172: PPC64: Support AES intrinsics
    • S8152730: File Leak in CompileBroker::init_compiler_thread_log of compileBroker.cpp:1665.
    • S8152981: Double icons with JMenuItem setHorizontalTextPosition on Win 10
    • S8153134: Infinite loop in handle_wrong_method in jmod
    • S8153176: ParOldGC: Use correct TaskQueueSet for ParallelTaskTerminator in marking.
    • S8153267: nmethod’s exception cache not multi-thread safe
    • S8153925: (fs) WatchService hangs on GetOverlappedResult and locks directory (win)
    • S8154945: Enable 8130150 and 8081778 intrinsics by default
    • S8155691: Update GIFlib library to the latest up-to-date
    • S8155781: C2: opaque unsafe access triggers an assert
    • S8156116: [macosx] two JNI locals to delete in AWTWindow.m, CGraphicsEnv.m
    • S8156723: JVM crash at sun.java2d.windows.GDIBlitLoops.nativeBlit
    • S8157035: Use stronger algorithms and keys for JSSE testing
    • S8157181: Compilers accept modification of final fields outside initializer methods
    • S8157184: java/lang/invoke/LFCaching/LFMultiThreadCachingTest.java failed with a fatal error
    • S8157548: JVM crashes sometimes while starting
    • S8157665: ProblemList.txt needs to be updated as 7041639 closed
    • S8158325: [macosx] Memory leak in com.apple.laf.ScreenMenu
    • S8158356: SIGSEGV when attempting to rotate BufferedImage using AffineTransform by NaN degrees
    • S8158639: C2 compilation fails with SIGSEGV
    • S8159035: CTSMode.java test crashed due to unhandled case of cipher length value as 0
    • S8159696: java.beans.MethodRef#get throws NullPointerException
    • S8160024: jdb returns invalid argument count if first parameter to Arrays.asList is null
    • S8160119: Utils.tryFindJvmPid sometimes find incorrect pid
    • S8160174: java.net.NetworkInterface – fixes and improvements for network interface listing
    • S8160370: System.getProperty(“os.version”) returns “Unknown” on Mac
    • S8160551: assert(c == Bytecodes::_putfield) failed: must be putfield
    • S8160570: [macosx] modal dialog can skip the activation/focus events
    • S8160623: [PIT] Exception running java/awt/event/KeyEvent/KeyChar/KeyCharTest.java
    • S8160664: JVM crashed with font manager on Solaris 12
    • S8160696: IllegalArgumentException: adding a component to a container on a different GraphicsDevice
    • S8160893: [macosx] JMenuItems in JPopupMenu are not accessible
    • S8160913: [TEST_BUG] javax/management/security/SecurityTest.java fails due to missing keyStore file
    • S8160941: “text/uri-list” dataflavor concats the first two strings
    • S8161039: System.getProperty(“os.version”) returns incorrect version number on Mac
    • S8161664: Memory leak in com.apple.laf.AquaProgressBarUI removed progress bar still referenced
    • S8161733: [parfait] Memory leak in imageioJPEG.c:2803
    • S8161742: [parfait] Uninitialised memory in isXTestAvailable of awt_Robot.c:65
    • S8161761: [parfait] 2 JNI exception pending defect groups in GraphicsPrimitiveMgr.c
    • S8162101: C2: Handle “wide” aliases for unsafe accesses
    • S8162488: JDK should be updated to use LittleCMS 2.8
    • S8162496: missing precedence edge for anti_dependence
    • S8162598: XSLTC transformer swallows empty namespace declaration which is needed to undeclare default namespace
    • S8162603: Unrecognized VM option ‘UseCountedLoopSafepoints’
    • S8162766: Unsafe_DefineClass0 accesses raw oops while in _thread_in_native
    • S8162795: [REDO] MemberNameTable doesn’t purge stale entries
    • S8162796: [macosx] LinearGradientPaint and RadialGradientPaint are not printed on OS X.
    • S8163104: Unexpected NPE still possible on some Kerberos ticket calls
    • S8163181: Further improvements for Unix NetworkInterface native implementation
    • S8163518: Integer overflow in StringBufferInputStream.read() and CharArrayReader.read/skip()
    • S8163877: Tests added in JDK-8163518 fail on some platforms
    • S8163896: Finalizing one key of a KeyPair invalidates the other key
    • S8163945: Honor Number type hint in toPrimitive on Numbers
    • S8163969: Cyclic interface initialization causes JVM crash
    • S8163979: [macosx] Chinese text shows as Latin w/ openVanilla input method
    • S8164002: Add a new CPU family (S_family) for SPARC S7 and above processors
    • S8164119: MoveToOtherScreenTest fails due wrong key attribute
    • S8164366: ZoneOffset.ofHoursMinutesSeconds() does not reject invalid input
    • S8164501: Uninitialised memory in byteArrayToPacket of SharedMemoryConnection.c
    • S8164508: unexpected profiling mismatch in c1 generated code
    • S8164784: ISO 4217 amendment 162
    • S8164846: CertificateException missing cause of underlying exception
    • S8165153: Crash in rebuild_cpu_to_node_map
    • S8165243: Base64.Encoder.wrap(os).write(byte[],int,int) with incorrect arguments should not produce output
    • S8165320: Small flaw when integrating 8160174 to JDK8
    • S8165428: Security Warning dialog should be always on the top when multiple applets with APPLICATION_MODAL dialog launched in a browser
    • S8165482: java in ldoms, with cpu-arch=generic has problems
    • S8165717: [macosx] Various memory leaks in jdk9
    • S8165751: NPE hit with java.security.debug=provider
    • S8165829: Android Studio 2.x crashes with NPE at sun.lwawt.macosx.CAccessibility.getAccessibleIndexInParent
    • S8165988: Test JarURLConnectionUseCaches.java fails at windows: failed to clean up files after test
    • S8166248: tools/pack200/Pack200Test.java fails on Win32: Could not reserve enough space
    • S8166507: ConcurrentSkipListSet.clear() can leave the Set in an invalid state
    • S8166747: Add invalid network / computer name cases to isReachable known failure switch
    • S8166799: ASSEMBLY_EXCEPTION contains historical company name
    • S8166840: Synthetic bridge constructor in ArrayList$Itr blocks inlining
    • S8166872: GPL header in /hotspot/src/share/vm/gc_implementation/g1/g1RemSetSummary.cpp
    • S8166902: Nested object literal property maps not reset in optimistic recompilation
    • S8167102: [macosx] PrintRequestAttributeSet breaks page size set using PageFormat
    • S8168115: Remove and retag jdk8u122-b03 in 8u repo
    • S8168364: [macosx] Delete unused class NSPrintinfo
    • S8168373: don’t emit conversions for symbols outside their lexical scope
    • S8168405: Pending exceptions in java.base/windows/native
    • S8168751: Two “Direct Clip” threads are created to play the same “AudioClip” object, what makes clip sound corrupted
    • S8168914: Crash in ClassLoaderData/JNIHandleBlock::oops_do during concurrent marking
    • S8169056: StringIndexOutOfBoundsException in Pattern.compile with CANON_EQ flag
    • S8169112: java.lang.VerifyError: (class: GregorSamsa, method: template-bash signature: (LGregorSamsa8;)V) Register 10 contains wrong type
    • S8169229: RSAClientKeyExchange debug info is incorrect
    • S8169355: Diacritics input works incorrectly on Windows if Spanish (Latin American) keyboard layout is used
    • S8169481: StackOverflowError on start when parsing PAC file to autodetect Proxy settings
    • S8169556: Wrapping of FileInputStream’s native skip and available methods
    • S8169865: Downport minor fixes in java.net native code from JDK 9 to JDK 8
    • S8170153: PPC64/s390x/aarch64: Poor StrictMath performance due to non-optimized compilation
    • S8170278: ticket renewal won’t happen with debugging turned on
    • S8170409: CMS: Crash in CardTableModRefBSForCTRS::process_chunk_boundaries
    • S8170465: JNI exception pending in jni_util.c:190
    • S8170466: JNI exception pending in jni_util.c:190
    • S8170536: Uninitialised memory in set_uintx_flag of attachListener.cpp
    • S8170552: [macosx] Wrong rendering of diacritics on macOS
    • S8170565: JSObject call() is passed undefined for the argument ‘thiz’
    • S8170578: CUPS Printing is broken with Ubuntu 16.10 (CUPS 2.2)
    • S8170594: >>>=0 generates invalid bytecode for BaseNode LHS
    • S8170913: Java “1.8.0_112″ on Windows 10 displays different characters for EUDCs from ones created in eudcedit.exe.
    • S8170950: Text is displayed in bold when fonts are installed into symlinked folder
    • S8170977: SparseArrayData should not grow its underlying dense array data
    • S8171151: JDK8u ProblemList.txt Updates
    • S8171155: Scanning method file for initialized final field updates can fail for non-existent fields
    • S8171194: Exception “Duplicate field name&signature in class file” should report the name and signature of the field
    • S8171219: Missing checks in sparse array shift() implementation
    • S8171808: Performance problems in dialogs with large tables when JAB activated
    • S8171849: Collection and Queue conversions not prioritized for Arrays
    • S8171909: [PIT] on Windows, failure of java/awt/Dialog/DialogAboveFrame/DialogAboveFrameTest.java
    • S8171952: [macosx] AWT_Modality/Automated/ModalExclusion/NoExclusion/ModelessDialog test fails as DummyButton on Dialog did not gain focus when clicked.
    • S8172053: (ppc64) Downport of 8170153 breaks build on linux/ppc64 (big endian)
    • S8172145: C2: anti dependence missed because store hidden by membar
    • S8173373: C1: NPE is thrown instead of LinkageError when accessing inaccessible field on NULL receiver
    • S8173423: Wrong display name for supplemental Japanese era
    • S8173581: performance regression in com/sun/crypto/provider/OutputFeedback.java
    • S8173654: Regression since 8u60: System.getenv doesn’t return env var set in JNI code
    • S8173664: Typo in https://java.net/downloads/heap-snapshot/hprof-binary-format.html
    • S8173853: IllegalArgumentException in java.awt.image.ReplicateScaleFilter
    • S8173876: Fast precise scrolling and DeltaAccumulator fix for macOS Sierra 10.12.2
    • S8174736: [JCP] [Mac]Cannot launch JCP on Mac os with language set to “Chinese, Simplified” while region is not China
    • S8174779: Locale issues with Mac 10.12
    • S8175025: The copyright section in the test/java/awt/font/TextLayout/DiacriticsDrawingTest.java should be updated
    • S8175131: sun.rmi.transport.tcp.TCPChannel.createConnection close connection on timeout
    • S8175345: Reported null pointer dereference defect groups
    • S8175915: NullPointerException from JComboBox and JList when Accessibility enabled
    • S8176287: [macosx] The print test crashed with Nimbus L&F
    • S8176490: [macosx] Sometimes NSWindow.isZoomed hangs
    • S8176511: JSObject property access is broken for numeric keys outside the int range
    • S8176530: JDK support for JavaFX modal print dialogs
    • S8177095: Range check dependent CastII/ConvI2L is prematurely eliminated
    • S8177450: javax.swing.text.html.parser.Parser parseScript ignores a character after commend end
    • S8177678: Overstatement of universality of Era.getDisplayName() implementation
    • S8177776: Create an equivalent test case for JDK9′s SupplementalJapaneseEraTest
    • S8177817: Remove assertions in 8u that were removed by 8056124 in 9.
    • S8179515: Class java.util.concurrent.ThreadLocalRandom fails to Initialize when using SecurityManager
    • S8180511: Null pointer dereference in Matcher::ReduceInst()
    • S8180565: Null pointer dereferences of ConstMethod::method()
    • S8180575: Null pointer dereference in LoadNode::Identity()
    • S8180576: Null pointer dereference in Matcher::xform()
    • S8180617: Null pointer dereference in InitializeNode::complete_stores
    • S8180660: missing LNT entry for finally block
    • S8180813: Null pointer dereference of CodeCache::find_blob() result
    • S8180934: PutfieldError failed with UnsupportedClassVersionError
    • S8180949: Correctly handle exception in TCPChannel.createConnection
    • S8181191: getUint32 returning Long
    • S8181192: [macos] javafx.print.PrinterJob.showPrintDialog() hangs on macOS
    • S8181205: JRE fails to load/register security providers when started from UNC pathname
    • S8181500: [TESTBUG] com/sun/jdi/LineNumberInfo.java fails with jArrayIndexOutOfBoundsException
    • S8182672: Java 8u121 on Linux intermittently returns null for MAC address
    • S8185779: 8u152 L10n resource file update
    • S8186532: 8u152 L10n resource file update md20
  • Import of OpenJDK 8 u161 build 12
    • S8035105: DNS provider cleanups
    • S8137255: sun/security/provider/NSASuiteB/TestDSAGenParameterSpec.java timeouts intermittently
    • S8140436: Negotiated Finite Field Diffie-Hellman Ephemeral Parameters for TLS
    • S8148108: Disable Diffie-Hellman keys less than 1024 bits
    • S8148421, PR3504: Transport Layer Security (TLS) Session Hash and Extended Master Secret Extension
    • S8154344: sun/security/pkcs11/KeyAgreement/SupportedDHKeys.java fails on solaris
    • S8156502: Use short name of SupportedEllipticCurvesExtension.java
    • S8158116: com/sun/crypto/provider/KeyAgreement/SupportedDHParamGens.java failed with timeout
    • S8159240: XSOM parser incorrectly processes type names with whitespaces
    • S8159377: JMX Connections need white-list filters
    • S8161598: Kitchensink fails: assert(nm->insts_contains(original_pc)) failed: original PC must be in nmethod/CompiledMethod
    • S8163237: Restrict the use of EXPORT cipher suites
    • S8166362: [TEST_BUG] test sun/net/www/http/HttpClient/B8025710.java failing with cert error in 8u121 b01
    • S8169944: sun/security/tools tests fail with CompilationError
    • S8170157: Enable unlimited cryptographic policy by default in OracleJDK
    • S8170245: [TEST_BUG] Cipher tests fail when running with unlimited policy
    • S8170911: 8170823 changes in 8u131 causing signature-test failure in jck runtime8b suite
    • S8171262: Stability fixes for lcms
    • S8172297: In java 8, the marshalling with JAX-WS does not escape carriage return
    • S8173631: Backout three hotspot fixes from 8u121-bpr repo
    • S8176536: Improved algorithm constraints checking
    • S8177144: [TEST BUG] sun/net/www/http/HttpClient/B8025710.java should run in ovm mode
    • S8177750: Forgot TestSocketFactory testcase in 8163958 changeset
    • S8178536: OOM ERRORS + SERVICE-THREAD TAKES A PROCESSOR TO 100%
    • S8178728: Check the AlgorithmParameters in algorithm constraints
    • S8179223: Integrate 8176490 into 8u131-bpr repo
    • S8181439: Test the jdk.tls.namedGroups System Property
    • S8181975: Run sun/security/pkcs11 tests on Mac
    • S8182614: Backout JDK-8140436 from 8u161
    • S8183939: Import JDK-8182672 into 8u141-bpr repo
    • S8184016: Text in native popup is not always updated with Sogou IME
    • S8185572: Enable AssumeMP by default on SPARC machines
    • S8185628: Backport jdk/test/lib/testlibrary/CompilerUtils.java to jdk8u which is helpful in test development
    • S8185719: rmi TestSocketFactory does not flush
    • S8185909: Disable JARs signed with DSA keys less than 1024 bits
    • S8186539: [testlibrary] TestSocketFactory should allow triggers before match/replace
    • S8187482: Backout JDK-8159377
    • S8187556: Backout of a fix reintroduced a dependency that had since been removed
    • S8188880: A JAXB JCK test failure found after 8186080
    • S8190258, PR3499: (tz) Support tzdata2017c
    • S8190259, PR3499: test tck.java.time.zone.TCKZoneRules is broken by tzdata2017c
    • S8190266: closed/java/awt/ComponentOrientation/WindowTest.java throws java.util.MissingResourceException.
    • S8190449: sun/security/pkcs11/KeyPairGenerator/TestDH2048.java fails on Solaris x64 5.10
    • S8190497: DHParameterSpec.getL() returns zero after JDK-8072452
    • S8190541: 8u161 L10n resource file update
    • S8190789: sun/security/provider/certpath/LDAPCertStore/TestURICertStoreParameters.java fails after JDK-8186606
    • S8191137, PR3503: keytool fails to format resource strings for keys for some languages after JDK-8171319
    • S8191340: 8ux linux builds failing – com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl (unknown type)
    • S8191352: Remove 8u161-b07 and 8u162-b07 tag to include 8191340 for b07
    • S8191907: PPC64 part of JDK-8174962: Better interface invocations
    • S8192793: 8u161 L10n resource file update md20
    • S8193683: Increase the number of clones in the CloneableDigest
  • Backports
    • S8165996, PR3506: PKCS11 using NSS throws an error regarding secmod.db when NSS uses sqlite
    • S8170523, PR3506: Some PKCS11 test cases are ignored with security manager
    • S8187667, PR3517: Disable deprecation warning for readdir_r
    • S8189789, PR3530: tomcat gzip-compressed response bodies appear to be broken in update 151
    • S8194739, PR3520: Zero port of 8174962: Better interface invocations
    • S8194828, PR3520: Fix for 8194739: Zero port of 8174962: Better interface invocations
    • S8195837, PR3522: (tz) Upgrade time-zone data to tzdata2018c
    • S8196516, PR3523: [REDO] [linux] libfontmanager should be linked against headless awt library
    • S8196978, PR3517: JDK-8187667 fails on GCC 4.4.7 as found on RHEL 6
  • Bug fixes
    • PR3489: Incomplete reading of directory containing tzdb.dat
    • PR3501: Revert 8165320 so as not to revert the work of 8034174 (PR2290)
    • PR3526: configure fails to parse GCC version
  • AArch64 port
    • S8164113, PR3519: AArch64: follow-up the fix for 8161598
    • S8193133, PR3519: Assertion failure because 0xDEADDEAD can be in-heap
    • S8194686, PR3519: AArch64 port of 8174962: Better interface invocations
    • S8195859, PR3519: AArch64: vtableStubs gtest fails after 8174962
    • S8196136, PR3519: AArch64: Correct register use in patch for JDK-8194686
    • S8196221, PR3519: AArch64: Mistake in committed patch for JDK-8195859
    • PR3519: Fix functions with missing return value.
    • PR3519: Fix further functions with a missing return value.
  • AArch32 port
    • S8140584: [aarch32] nmethod::oops_do_marking_epilogue always runs verification code
    • S8174962: [aarch32] Better interface invocations
  • Shenandoah
    • Fix post-CPU merge Shenandoah regressions

The tarballs can be downloaded from:

We provide both gzip and xz tarballs, so that those who are able to make use of the smaller tarball produced by xz may do so.

The tarballs are accompanied by digital signatures available at:

These are produced using my public key. See details below.

  • PGP Key: ed25519/0xCFDA0F9B35964222 (hkp://keys.gnupg.net)
  • Fingerprint = 5132 579D D154 0ED2 3E04 C5A0 CFDA 0F9B 3596 4222

GnuPG >= 2.1 is required to be able to handle this key.

SHA256 checksums:

  • 936302694e193791885e81cf72097eeadee5b68ba220889228b0aafbfb2cb654 icedtea-3.7.0.tar.gz
  • 1361859f0436dd542bb3aa0e1441331158217fcc4e12ead9c1b23654811cf7ea icedtea-3.7.0.tar.gz.sig
  • bc50a4df0e9f81cb58e0e0cbd10bc7efda4f7ff11026b43d3b9ab19b91fdd827 icedtea-3.7.0.tar.xz
  • 4c252427ca9164663e4090279143a28889adf8cf6fd8240b6b34a023a71b4eb3 icedtea-3.7.0.tar.xz.sig

The checksums can be downloaded from:

A 3.7.0 ebuild for Gentoo is available.

The following people helped with these releases:

We would also like to thank the bug reporters and testers!

To get started:

$ tar xzf icedtea-3.7.0.tar.gz

or:

$ tar x -I xz -f icedtea-3.7.0.tar.xz

then:

$ mkdir icedtea-build
$ cd icedtea-build
$ ../icedtea-3.7.0/configure
$ make

Full build requirements and instructions are available in the INSTALL file.

Happy hacking!

The IcedTea project provides a harness to build the source code from OpenJDK using Free Software build tools, along with additional features such as the ability to build against system libraries and support for alternative virtual machines and architectures beyond those supported by OpenJDK.

This release updates our OpenJDK 7 support in the 2.6.x series with the January 2018 security fixes from OpenJDK 7 u171.

If you find an issue with the release, please report it to our bug database under the appropriate component. Development discussion takes place on the distro-pkg-dev OpenJDK mailing list and patches are always welcome.

Full details of the release can be found below.

What’s New?

New in release 2.6.13 (2018-02-27)

  • Security fixes
  • Import of OpenJDK 7 u171 build 0
    • S7171982: Cipher getParameters() throws RuntimeException: Cannot find SunJCE provider
    • S7172652: With JDK 1.7 text field does not obtain focus when using mnemonic Alt/Key combin
    • S8022532: [parfait] Potential memory leak in gtk2_interface.c
    • S8031003: [Parfait] warnings from jdk/src/share/native/sun/security/jgss/wrapper: JNI exception pending
    • S8035105: DNS provider cleanups
    • S8041781: Need new regression tests for PBE keys
    • S8041787: Need new regressions tests for buffer handling for PBE algorithms
    • S8044193: Need to add known answer tests for AES cipher
    • S8048601: Tests for JCE crypto ciphers (part 1)
    • S8048819: Implement reliability test for DH algorithm
    • S8072452: Support DHE sizes up to 8192-bits and DSA sizes up to 3072-bits
    • S8075286: Additional tests for signature algorithm OIDs and transformation string
    • S8137255: sun/security/provider/NSASuiteB/TestDSAGenParameterSpec.java timeouts intermittently
    • S8141243: Unexpected timezone returned after parsing a date
    • S8144593: Suppress not recognized property/feature warning messages from SAXParser
    • S8147969: Print size of DH keysize when errors are encountered
    • S8148108: Disable Diffie-Hellman keys less than 1024 bits
    • S8148421, PR3505: Transport Layer Security (TLS) Session Hash and Extended Master Secret Extension
    • S8154344: sun/security/pkcs11/KeyAgreement/SupportedDHKeys.java fails on solaris
    • S8156502: Use short name of SupportedEllipticCurvesExtension.java
    • S8157548: JVM crashes sometimes while starting
    • S8157603: TestCipher.java doesn’t check one of the decrypted message as expected
    • S8158116: com/sun/crypto/provider/KeyAgreement/SupportedDHParamGens.java failed with timeout
    • S8159240: XSOM parser incorrectly processes type names with whitespaces
    • S8163237: Restrict the use of EXPORT cipher suites
    • S8163958: Improved garbage collection [test case]
    • S8166248: tools/pack200/Pack200Test.java fails on Win32: Could not reserve enough space
    • S8166362: [TEST_BUG] test sun/net/www/http/HttpClient/B8025710.java failing with cert error in 8u121 b01
    • S8170157: Enable unlimited cryptographic policy by default in OracleJDK
    • S8170245: [TEST_BUG] Cipher tests fail when running with unlimited policy
    • S8170536: Uninitialised memory in set_uintx_flag of attachListener.cpp
    • S8177144: [TEST BUG] sun/net/www/http/HttpClient/B8025710.java should run in ovm mode
    • S8178728: Check the AlgorithmParameters in algorithm constraints
    • S8180048: Interned string and symbol table leak memory during parallel unlinking
    • S8184016: Text in native popup is not always updated with Sogou IME
    • S8185628: Backport jdk/test/lib/testlibrary/CompilerUtils.java to jdk8u which is helpful in test development
    • S8185719: rmi TestSocketFactory does not flush
    • S8185909: Disable JARs signed with DSA keys less than 1024 bits
    • S8186539: [testlibrary] TestSocketFactory should allow triggers before match/replace
    • S8187667, PR3518: Disable deprecation warning for readdir_r
    • S8188880: A JAXB JCK test failure found after 8186080
    • S8190258, PR3500: (tz) Support tzdata2017c
    • S8190259, PR3500: test tck.java.time.zone.TCKZoneRules is broken by tzdata2017c
    • S8190266: closed/java/awt/ComponentOrientation/WindowTest.java throws java.util.MissingResourceException.
    • S8190449: sun/security/pkcs11/KeyPairGenerator/TestDH2048.java fails on Solaris x64 5.10
    • S8190497: DHParameterSpec.getL() returns zero after JDK-8072452
    • S8190541: 8u161 L10n resource file update
    • S8190789: sun/security/provider/certpath/LDAPCertStore/TestURICertStoreParameters.java fails after JDK-8186606
    • S8192793: 8u161 L10n resource file update md20
    • S8193683: Increase the number of clones in the CloneableDigest
    • S8194859: Bad backport of 8024468 breaks Zero build due to lack of 8010862 in OpenJDK 7
    • S8195837: (tz) Upgrade time-zone data to tzdata2018c
  • Import of OpenJDK 7 u171 build 1
    • S8007772: G1: assert(!hr->isHumongous() || mr.start() == hr->bottom()) failed: the start of HeapRegion and MemRegion should be consistent for humongous regions
    • S8022956: Clang: enable return type warnings on BSD
    • S8043029: Change 8037816 breaks HS build with older GCC versions which don’t support diagnostic pragmas
    • S8048169: Change 8037816 breaks HS build on PPC64 and CPP-Interpreter platforms
    • S8062808: Turn on the -Wreturn-type warning
    • S8064786: Fix debug build after 8062808: Turn on the -Wreturn-type warning
    • S8143245: Zero build requires disabled warnings
    • S8196952, PR3525: Bad primeCertainty value setting in DSAParameterGenerator
    • S8196978: JDK-8187667 fails on GCC 4.4.7 as found on RHEL 6
    • S8197510: fastdebug builds fail due to lack of p2i
    • S8197801: Zero debug build fails on “assert(labs(istate->_stack_base – istate->_stack_limit) == (istate->_method->max_stack() + extra_stack_entries + 1)) failed: bad stack limit”
  • Import of OpenJDK 7 u171 build 2
    • S8197981: Missing return statement in __sync_val_compare_and_swap_8
  • Backports
    • S7189886, PR3507: (aio) Add test coverage for AsynchronousChannelGroup.withThreadPool
    • S7200306, PR3507: SunPKCS11 provider delays the check of DSA key size for SHA1withDSA to sign() instead of init()
    • S8012930, PR3507: (fs) Eliminate recursion from FileTreeWalker
    • S8013647, PR3507: JPRT unable to clean-up after tests that leave file trees with loops
    • S8020321, PR3507: Problem in PKCS11 regression test TestRSAKeyLength
    • S8022313, PR3507: sun/security/pkcs11/rsa/TestKeyPairGenerator.java failed in aurora
    • S8027218, PR3507: TEST_BUG: sun/security/pkcs11/ec tests fail because of ever-changing key size restrictions
    • S8029158, PR3507: sun/security/pkcs11/Signature/TestDSAKeyLength.java does not compile (or run)
    • S8031113, PR3507: TEST_BUG: java/nio/channels/AsynchronousChannelGroup/Basic.java fails intermittently
    • S8048603, PR3507: Additional tests for MAC algorithms
    • S8048622, PR3507: Enhance tests for PKCS11 keystores with NSS
    • S8075565, PR3337: Define @intermittent jtreg keyword and mark intermittently failing jdk tests
    • S8075670, PR3507: Remove intermittent keyword from some tests
    • S8078334, PR3507: Mark regression tests using randomness
    • S8078880, PR3507: Mark a few more intermittently failuring security-libs
    • S8133318, PR3507: Exclude intermittent failing PKCS11 tests on Solaris SPARC 11.1 and earlier
    • S8144539, PR3507: Update PKCS11 tests to run with security manager
    • S8151731, PR3337: Add new jtreg keywords to jdk 8
    • S8165996, PR3507: PKCS11 using NSS throws an error regarding secmod.db when NSS uses sqlite
    • S8170523, PR3507: Some PKCS11 test cases are ignored with security manager
    • S8196516, PR3524: [REDO] [linux] libfontmanager should be linked against headless awt library
  • AArch64 port
    • S8193133, PR3521: Assertion failure because 0xDEADDEAD can be in-heap
    • PR3521: Fix functions with missing return value.
    • PR3521: Fix further functions with a missing return value.

The tarballs can be downloaded from:

We provide both gzip and xz tarballs, so that those who are able to make use of the smaller tarball produced by xz may do so.

The tarballs are accompanied by digital signatures available at:

These are produced using my public key. See details below.

  • PGP Key: ed25519/0xCFDA0F9B35964222 (hkp://keys.gnupg.net)
  • Fingerprint = 5132 579D D154 0ED2 3E04 C5A0 CFDA 0F9B 3596 4222

GnuPG >= 2.1 is required to be able to handle this key.

SHA256 checksums:

  • 7528c905da9b1c09aef16a938e7d879c8cdb14f93b1a25f0ec041307572c3d4f icedtea-2.6.13.tar.gz
  • f4f98da8894fbcf9e55a1a5cec9f23b6281aa0cb4cc1714be61b0a25e916383f icedtea-2.6.13.tar.gz.sig
  • 104e84205d1176e217e24f770784c53d1cd666aeb23ab0bae8ac858e5b0e63f0 icedtea-2.6.13.tar.xz
  • 7e81a4d785484e1dbc504ca418be84d8393b5d790cc11f1cd61c8f6cefa0543c icedtea-2.6.13.tar.xz.sig

The checksums can be downloaded from:

A 2.6.13 ebuild for Gentoo is available.

The following people helped with these releases:

We would also like to thank the bug reporters and testers!

To get started:

$ tar xzf icedtea-2.6.13.tar.gz

or:

$ tar x -I xz -f icedtea-2.6.13.tar.xz

then:

$ mkdir icedtea-build
$ cd icedtea-build
$ ../icedtea-2.6.13/configure
$ make

Full build requirements and instructions are available in the INSTALL file.

Happy hacking!

At Fosdem we had a talk on dtrace for linux in the Debugging Tools devroom.

Not explicitly mentioned in that talk, but certainly the most exciting thing, is that Oracle is doing a proper linux kernel port:

 commit e1744f50ee9bc1978d41db7cc93bcf30687853e6
 Author: Tomas Jedlicka <tomas.jedlicka@oracle.com>
 Date: Tue Aug 1 09:15:44 2017 -0400

 dtrace: Integrate DTrace Modules into kernel proper

 This changeset integrates DTrace module sources into the main kernel
 source tree under the GPLv2 license. Sources have been moved to
 appropriate locations in the kernel tree.

That is right, dtrace dropped the CDDL and switched to the GPL!

The user space code dtrace-utils and libdtrace-ctf (a combination of GPLv2 and UPL) can be found on the DTrace Project Source Control page. The NEWS file mentions the license switch (and that it is build upon elfutils, which I personally was pleased to find out).

The kernel sources (GPLv2+ for the core kernel and UPL for the uapi) are slightly harder to find because they are inside the uek kernel source tree, but following the above commit you can easily get at the whole linux kernel dtrace directory.

Update: There is now a dtrace-linux-kernel.git repository with all the dtrace commits rebased on top of recent upstream linux kernels.

The UPL is the Universal Permissive License, which according to the FSF is a lax, non-copyleft license that is compatible with the GNU GPL.

Thank you Oracle for making everyone’s life easier by waving your magic relicensing wand!

Now there is lots of hard work to do to actually properly integrate this. And I am sure there are a lot of technical hurdles when trying to get this upstreamed into the mainline kernel. But that is just hard work. Which we can now start collaborating on in earnest.

Like systemtap and the Dynamic Probes (dprobes) before it, dtrace is a whole system observability tool combining tracing, profiling and probing/debugging techniques. Something the upstream linux kernel hackers don’t always appreciate when presented as one large system. They prefer having separate small tweaks for tracing, profiling and probing which are mostly separate from each other. It took years for the various hooks, kprobes, uprobes, markers, etc. from systemtap (and other systems) to get upstream. But these days they are. And there is now even a byte code interpreter (eBPF) in the mainline kernel as originally envisioned by dprobes, which systemtap can now target through stapbpf. So with all those techniques now available in the linux kernel it will be exciting to see if dtrace for linux can unite them all.

There have been a few efforts at writing an Emacs JIT — the original one, Burton Samograd’s, and also Nick LLoyd’s. So, what else to do except write my own?

Like the latter two, I based mine on GNU libjit. I did look at a few other JIT libraries: LLVM, gcc-jit, GNU Lightning, MyJit.  libjit seemed like a nice middle ground between a JIT with heavy runtime costs (LLVM, GCC) and one that is too lightweight (Lightning).

All of these Emacs JITs work by compiling bytecode to native code.  Now, I don’t actually think that is the best choice — it’s just the easiest — but my other project to do a more complete job in this area isn’t really ready to be released.  So bytecode it is.

Emacs implements a somewhat weird stack-based bytecode.  Many ordinary things are there, but seemingly obvious stack operations like “swap” do not exist; and there are bytecodes for very specialized Emacs operations like forward-char or point-max.

Samograd describes his implementation as “compiling down the spine”.  What he means by this is that the body of each opcode is implemented by some C function, and the JIT compiler emits, essentially, a series of subroutine calls.  This used to be called “jsr threading” in the olden days, though maybe it has some newer names by now.

Of course, we can do better than this, and Lloyd’s JIT does.  His emits instructions for the bodies of most bytecodes, deferring only a few to helper functions.  This is a better approach because many of these operations are only one or two instructions.

However, his approach takes a wrong turn by deferring stack operations to the compiled code.  For example, in this JIT, the Bdiscard opcode, which simply drops some items from the stack, is implemented as:

 CASE (Bdiscard):
 {
   JIT_NEED_STACK;
   JIT_INC (ctxt.stack, -sizeof (Lisp_Object));
   JIT_NEXT;
   NEXT;
 }

It turns out, though, that this isn’t needed — at least, for the bytecode generated by the Emacs byte-compiler, the stack depth at any given PC is a constant.  This means that the stack adjustments can be done at compile time, not runtime, leading to a performance boost.  So, the above opcode doesn’t need to emit code at all.

(And, if you’re worried about hand-crafted bytecode, it’s easy to write a little bytecode verifier to avoid JIT-compiling invalid things.  Though of course you shouldn’t worry, since you can already crash Emacs with bad bytecode.)

So, naturally, my implementation does not do this extra work.  And, it inlines more operations besides.

Caveat

I’ve only enabled the JIT for bytecode that uses lexical binding.  There isn’t any problem enabling it everywhere, I just figured it probably isn’t that useful, and so I didn’t bother.

Results

The results are pretty good.  First of all, I have it set up to automatically JIT compile every function, and this doesn’t seem any slower than ordinary Emacs, and it doesn’t crash.

Using the “silly-loop” example from the Emacs Lisp manual, with lexical binding enabled, I get these results:

Mode Time
Interpreted 4.48
Byte compiled 0.91
JIT 0.26

This is essentially the best case for this JIT, though.

Future Directions

I have a few ideas for how to improve the performance of the generated code.  One way to look at this is to look at Emacs’ own C code, to see what advantages it has over JIT-compiled code.  There are really three: cheaper function calls, inlining, and unboxing.

Calling a function in Emacs Lisp is quite expensive.  A call from the JIT requires marshalling the arguments into an array, then calling Ffuncall; which then might dispatch to a C function (a “subr”), the bytecode interpreter, or the ordinary interpreter.  In some cases this may require allocation.

This overhead applies to nearly every call — but the C implementation of Emacs is free to call various primitive functions directly, without using Ffuncall to indirect through some Lisp symbol.

Now, these direct calls aren’t without a cost: they prevent the modification of some functions from Lisp.  Sometimes this is a pain (it might be handy to hack on load from Lisp), but in many cases it is unimportant.

So, one idea for the JIT is to keep a list of such functions and then emit direct calls rather than indirect ones.

Even better than this would be to improve the calling convention so that all calls are less expensive.  However, because a function can be redefined with different arguments, it is tricky to see how to do this efficiently.

In the Emacs C code, many things are inlined that still aren’t inlined in the JIT — just look through lisp.h for all the inline functions (and/or macros, lisp.h is “unusual”).  Many of these things could be done in the JIT, though in some cases it might be more work than it is worth.

Even better, but also even more difficult, would be inlining from one bytecode function into another.  High-performance JITs do this when they notice a hot spot in the code.

Finally, unboxing.  In the Emacs C code, it’s relatively normal to type-check Lisp objects and then work solely in terms of their C analogues after that point.  This is more efficient because it hoists the tag manipulations.  Some work like this could be done automatically, by writing optimization passes for libjit that work on libjit’s internal representation of functions.

Getting the Code

The code is on the libjit branch in my Emacs repository on github.  You’ll have to build your own libjit, too, and if you want to avoid hacking on the Emacs Makefile, you will need my fork of libjit that adds pkg-config files.

Maintenance of an aging Bugzilla instance is a burden, and since CACAO development has mostly migrated to Bitbucket, the bug tracker will be maintained there as well.

The new location for tracking bugs is Bitbucket issues.

All the historical content from Bugzilla is statically archived at this site on the Bugzilla page.

I always used Eclipse extensively, although I moved away from it when it started having all sort of rendering issues with RHEL, mostly when SWT moved to GTK3 underneath. Most of those problems are slowly being fixed, and the IDE is again very usable under RHEL. I promised Lars last year that I would start using Eclipse again once those problem were addressed, and here I am, keeping the promise!

One thing I never really totally enjoyed in Eclipse was the debugger, though. Of all the IDEs I tried, I find the NetBeans debugger the absolute best, followed by IntelliJ IDEA. I don’t think the Eclipse debugger is bad in itself, but it doesn’t directly do the things I expect by default, and I often need to do quite a bit of tweaking in order to get it right. I admit I’m not exactly the average developer though, so it’s possible that what NetBeans offers me simply matches more what I find most useful. For instance, the detailed view for variables and the fact that you can quickly attach to the native portion of a process (and this works with the JDK as well as any hybrid application). Eclipse can do that, but requires some fiddling (most of the manual process I described on this post).

Now, there are things I love about Eclipse too. For example, while is true that it doesn’t show the most detailed view of variables, you can quickly execute arbitrary code right during the debugging sessions, simply by writing it. I’m not sure about IDEA, but NetBeans has similar functionality, but it’s hidden under tons of menu that you need to click and configure, and for very complex stuff the best option is extending the debugger itself, which is not trivial.

Recently, I’ve been debugging a weird issue in Fedora, and I found I was in need to scan a very large Map of fonts. The default formatter made things quite complex to understand while all I wanted was to quickly see what was the closest key in the map to the one I had as input.

You can see what I mean in this very trivial example.

Here my map is a simple immutable one with just 3 values, however already here you can see that the comma used to separate the values in the variable view makes things quite complicated, just imagine if this map had some thousands values, sorting them would have been quite an experience!

        Map<String, String> map =
                Map.of("Test1", "1,2,2",
                       "Test2", "2,3,1",
                       "Test3", "3,2,1");

Produces:

{Test3=3,2,1, Test1=1,2,2, Test2=2,3,1}

Not quite what I want!

The IDE gives us a very powerful tool though. Instead of simply changing the formatter, we can execute code to analyse the code! There is an hidden gem called “Display” view. You find it under “Window > Show View”, in the Debug perspective it should be readily visible, otherwise you can select “Other” and bring all the views up.

This neat view is simply a secondary editor that is active only during a debugging session, and allows you to instrument the code on the fly. What this means, is that you can type the following in the Display view and scan all the values for Map in question:

for (String key : map.keySet()) {
    System.err.println("* " + key + " -> " + map.get(key));
}

Then simply press “Execute selected text” and the code will execute. The subtle difference between “Execute” and “Display Result”, which is the icon next to the execute, is that the result of the code snippet will be printed in the Display, this is the result of the execution of the code itself, so in our case the snippet executes without a result, which is considered “false” by Eclipse apparently.

but they both have side effects on the running process, so for instance if you modify the map (well, that one is immutable, but you get the point) both options will modify it, no matter where the result gets printed. The Display doesn’t seem to like lambdas or default methods, so you need to keep things a bit old fashioned, but that’s hardly a problem.

Overall I find this feature extremely useful, and worth the hassle of dealing with the default configurations on the Eclipse Debugger. Just be careful though, side effects in debugging are hidden at every step, sometime even just a toString() called by the debugger can change your code in unexpected ways!


I attended the 5th Chrome Dev Summit this week. The talks were all recorded and are available via the schedule (the keynote and leadership panel on day 1 are perhaps of broadest interest and highest bang-for-buck viewing value). It was a high quality, well-produced event with an intimate feel – I was very surprised when Robert Nyman told me it was over 700 people! I appreciated the good vegetarian food options and noticed and was very impressed by the much-better-than-typical-tech-conferences gender representation and code of conduct visibility.

It doesn’t always look this way from the outside, but the various browser engine teams are more often than not working toward the same goals and in constant contact. For those who don’t know that, it was nice to see the shoutouts for other browsers and the use of Firefox’s new logo!

The focus of the event IMO was, as expected, the mobile Web. While the audience was Web developers, it was interesting to see what the Chrome team is focusing on. Some of the efforts felt like Firefox OS 4 years ago but I guess FxOS was just ahead of its time 😉

From my perspective, Firefox is in pretty good shape for supporting the things Chrome was promoting (Service Workers, Custom Elements and Shadow DOM, wasm, performance-related tooling, etc.). There are of course areas we can improve: further work on Fennec support for add-to-homescreen, devtools, and seeing a few things through to release (e.g. JS modules, Custom Elements, and Shadow DOM – work is underway and we’re hoping for soon!). Oh, and one notable exception to being aligned on things is the Network Information API that the Mozilla community isn’t super fond.

Other highlights for me personally included the Chrome User Experience Report (“a public dataset of key user experience metrics for popular origins on the web, as experienced by Chrome users under real-world conditions”) and the discussion about improving the developer experience for Web Workers.

It was great putting faces to names and enjoying sunny San Francisco (no, seriously, it was super sunny and hot). Thanks for the great show, Google!

It’s been a long road, but at last the puzzle is complete: Today we delivered Project Jigsaw for general use, as part of JDK 9.

Jigsaw enhances Java to support programming in the large by adding a module system to the Java SE Platform and to its reference implementation, the JDK. You can now leverage the key advantages of that system, namely strong encapsulation and reliable configuration, to climb out of JAR hell and better structure your code for reusability and long-term evolution.

Jigsaw also applies the module system to the Platform itself, and to the massive, monolithic JDK, to improve security, integrity, performance, and scalability. The last of these goals was originally intended to reduce download times and scale Java SE down to small devices, but it is today just as relevant to dense deployments in the cloud. The Java SE 9 API is divided into twenty-six standard modules; JDK 9 contains dozens more for the usual development and serviceability tools, service providers, and JDK-specific APIs. As a result you can now deliver a Java application together with a slimmed-down Java run-time system that contains just the modules that your application requires.

We made all these changes with a keen eye — as always — toward compatibility. The Java SE Platform and the JDK are now modular, but that doesn’t mean that you must convert your own code into modules in order to run on JDK 9 or a slimmed-down version of it. Existing class-path applications that use only standard Java SE 8 APIs will, for the most part, work without change.

Existing libraries and frameworks that depend upon internal implementation details of the JDK may require change, and they may cause warnings to be issued at run time until their maintainers fix them. Some popular libraries, frameworks, and tools — including Maven, Gradle, and Ant — were in this category but have already been fixed, so be sure to upgrade to the latest versions.

Looking ahead It’s been a long road to deliver Jigsaw, and I expect it will be a long road to its wide adoption — and that’s perfectly fine. Many developers will make use of the newly-modular nature of the platform long before they use the module system in their own code, and it will be easier to use the module system for new code rather than existing code.

Modularizing an existing software system can, in fact, be difficult. Sometimes it won’t be worth the effort. Jigsaw does, however, ease that effort by supporting both top-down and bottom-up migration to modules. You can thus begin to modularize your own applications long before their dependencies are modularized by their maintainers. If you maintain a library or framework then we encourage you to publish a modularized version of it as soon as possible, though not until all of its dependencies have been modularized.

Modularizing the Java SE Platform and the JDK was extremely difficult, but I’m confident it will prove to have been worth the effort: It lays a strong foundation for the future of Java. The modular nature of the platform makes it possible to remove obsolete modules and to deliver new yet non-final APIs in incubator modules for early testing. The improved integrity of the platform, enabled by the strong encapsulation of internal APIs, makes it easier to move Java forward faster by ensuring that libraries, frameworks, and applications do not depend upon rapidly-changing internal implementation details.

Learning more There are by now plenty of ways to learn about Jigsaw, from those of us who created it as well as those who helped out along the way.

If your time is limited, consider one or more of the following:

  • The State of the Module System is a concise, informal written overview of the module system. (It’s slightly out of date; I’ll update it soon.)

  • Make Way for Modules!, my keynote presentation at Devoxx Belgium 2015, packs a lot of high-level information into thirty minutes. I followed that up a year later with a quick live demo of Jigsaw’s key features.

  • Alex Buckley’s Modular Development with JDK 9, from Devoxx US 2017, covers the essentials in more depth, in just under an hour.

If you really want to dive in:

Comments, questions, and suggestions are welcome on the jigsaw-dev mailing list. (If you haven’t already subscribed to that list then please do so first, otherwise your message will be discarded as spam.)

Thanks! Project Jigsaw was an extended, exhilarating, and sometimes exhausting nine-year effort. I was incredibly fortunate to work with an amazing core team from pretty much the very beginning: Alan Bateman, Alex Buckley, Mandy Chung, Jonathan Gibbons, and Karen Kinnear. To all of you: My deepest thanks.

Key contributions later on came from Sundar Athijegannathan, Chris Hegarty, Lois Foltan, Magnus Ihse Bursie, Erik Joelsson, Jim Laskey, Jan Lahoda, Claes Redestad, Paul Sandoz, and Harold Seigel.

Jigsaw benefited immensely from critical comments and suggestions from many others including Jayaprakash Artanareeswaran, Paul Bakker, Martin Buchholz, Stephen Colebourne, Andrew Dinn, Christoph Engelbert, Rémi Forax, Brian Fox, Trisha Gee, Brian Goetz, Mike Hearn, Stephan Herrmann, Juergen Hoeller, Peter Levart, Sander Mak, Gunnar Morling, Simon Nash, Nicolai Parlog, Michael Rasmussen, John Rose, Uwe Schindler, Robert Scholte, Bill Shannon, Aleksey Shipilëv, Jochen Theodorou, Weijun Wang, Tom Watson, and Rafael Winterhalter.

To everyone who contributed, in ways large and small: Thank you!

Thanks to Alan Bateman and Alex Buckley
for comments on drafts of this entry.

The Talos II is now available for pre-order. It is the more affordable, more power-efficient successor to the Talos I machine I wrote about in a previous post.

This is a very affordable machine for how powerful it will be, and there are minimal mainboard + CPU + RAM bundles (e.g., this one), around which one can build a workstation with more readily-available parts. I’ve placed an order for one of the bundles, and will buy the chassis and GPU separately (mainly to avoid high cross-border shipping fees for the full workstation).

The Talos II is an important machine for Free Software, and will likely be RYF-certified by the FSF. Pre-orders end September 15th!

For over twenty years the Java SE Platform and the JDK have evolved in large, irregular, and somewhat unpredictable steps. Each feature release has been driven by one or a few significant features, and so the schedule of each release has been adjusted as needed — sometimes more than once! — in order to accommodate the development of those features.

This approach made it possible to deliver big new features at a high level of quality, after thorough review and testing by early adopters. The downside, however, was that smaller API, language, and JVM features could only be delivered when the big features were ready.

This was an acceptable tradeoff in the decades before and after the turn of the century, when Java competed with just a few platforms which evolved at a similar stately pace. Nowadays, however, Java competes with many platforms which evolve at a more rapid pace.

For Java to remain competitive it must not just continue to move forward — it must move forward faster.

Back on the train Five years ago I mused in this space on the tension between developers, who prefer rapid innovation, and enterprises, which prefer stability, and the fact that everyone prefers regular and predictable releases.

To address these differing desires I suggested, back then, that we switch from the historical feature-driven release model to a time-driven “train” model, with a feature release every two years. In this type of model the development process is a continuous pipeline of innovation that’s only loosely coupled to the actual release process, which itself has a constant cadence. Any particular feature, large or small, is merged only when it’s nearly finished. If a feature misses the current train then that’s unfortunate but it’s not the end of the world, since the next train will already be waiting and will also leave on schedule.

The two-year train model was appealing in theory, but proved unworkable in practice. We took an additional eight months for Java 8 in order to address critical security issues and finish Project Lambda, which was preferable to delaying Lambda by two years. We initially planned Java 9 as a two-and-a-half year release in order to include Project Jigsaw, which was preferable to delaying Jigsaw by an additional eighteen months, yet in the end we wound up taking an additional year and so Java 9 will ship this month, three and a half years after Java 8.

A two-year release cadence is, in retrospect, simply too slow. To achieve a constant cadence we must ship feature releases at a more rapid rate. Deferring a feature from one release to the next should be a tactical decision with minor inconveniences rather than a strategic decision with major consequences.

So, let’s ship a feature release every six months.

That’s fast enough to minimize the pain of waiting for the next train, yet slow enough that we can still deliver each release at a high level of quality.

Proposal Taking inspiration from the release models used by other platforms and by various operating-system distributions, I propose that after Java 9 we adopt a strict, time-based model with a new feature release every six months, update releases every quarter, and a long-term support release every three years.

  • Feature releases can contain any type of feature, including not just new and improved APIs but also language and JVM features. New features will be merged only when they’re nearly finished, so that the release currently in development is feature-complete at all times. Feature releases will ship in March and September of each year, starting in March of 2018.

  • Update releases will be strictly limited to fixes of security issues, regressions, and bugs in newer features. Each feature release will receive two updates before the next feature release. Update releases will ship quarterly in January, April, July, and October, as they do today.

  • Every three years, starting in September of 2018, the feature release will be a long-term support release. Updates for these releases will be available for at least three years and quite possibly longer, depending upon your vendor.

In this model the overall rate of change should be about the same as it is today; what’s different is that there will be many more opportunities to deliver innovation. The six-month feature releases will be smaller than the multi-year feature releases of the past, and therefore easier to adopt. Six-month feature releases will also reduce the pressure to backport new features to older releases, since the next feature release will never be more than six months away.

Developers who prefer rapid innovation, so that they can leverage new features in production as soon as possible, can use the most recent feature release or an update release thereof and move on to the next one when it ships. They can deliver an application in a Docker image, or other type of container package, along with the exact Java release on which the application will run. Since the application and the Java release can always be tested together, in a modern continuous-integration and continuous-deployment pipeline, it will be straightforward to move from one Java release to the next.

Enterprises that prefer stability, so that they can run multiple large applications on a single shared Java release, can instead use the current long-term support release. They can plan ahead to migrate from one long-term support release to the next, like clockwork, every three years.

To make it clear that these are time-based releases, and to make it easy to figure out the release date of any particular release, the version strings of feature releases will be of the form $YEAR.$MONTH. Thus next year’s March release will be 18.3, and the September long-term support release will be 18.9.

Implications This proposal will, if adopted, require major changes in how contributors in the OpenJDK Community produce the JDK itself; I’ve posted some initial thoughts as to how we might proceed there. It will be made easier if we can reduce the overhead of the Java Community Process, which governs the evolution of the Java SE Platform; my colleagues Brian Goetz and Georges Saab have already raised this topic with the JCP Executive Committee.

This proposal will, ultimately, affect every developer, user, and enterprise that relies upon Java. It will, if successful, help Java remain competitive — while maintaining its core values of compatibility, reliability, and thoughtful evolution — for many years to come.

Comments and questions are welcome, either on the OpenJDK general discussion list (please subscribe to that list in order to post to it) or on Twitter, with the hashtag #javatrain.

An update on my notes to compile NetBSD kernels and userland.


Build / update the tools:

-U : for unprivilged building
-u : to update
-m : to specify architecture

./build.sh -U -u tools

To cross compile, this is would enough:
./build.sh -U -m i386 -u tools
However, since I do want to build on the same computer and the build script would be confused, we add -T /usr/tools-${HOST_ARCH}-${TARGET_ARCH} and also separate the object dir with -O!

./build.sh -U -m i386 -u -O /usr/obj-amd64-i386 -T /usr/tools-amd64-i386 tools


Then we build the kernel

./build.sh -U kernel=CONFNAME

or for cross compilation:
./build.sh -U -O /usr/obj-amd64-i386 -T /usr/tools-amd64-i386 -m i386 -u kernel=GENERIC

The modules:
./build.sh -U -u modules installmodules=/

Now to build userland, including X11. I did not attempt to cross-build userland yet.

./build.sh -U -x -u distribution

./build.sh -U -x -u distribution install=/
After almost fifteen years I have decided to quit working on IKVM.NET. The decision has been a long time coming. Those of you that saw yesterday’s Twitter spat, please don’t assume that was the cause. It rather shared an underlying cause. I’ve slowly been losing faith in .NET. Looking back, I guess this process started with the release of .NET 3.5. On the Java side things don’t look much better. The Java 9 module system reminds me too much of the generics erasure debacle.

I hope someone will fork IKVM.NET and continue working on it. Although, I’d appreciate it if they’d pick another name. I’ve gotten so much criticism for the name over the years, that I’d like to hang on to it 😊

I’d like to thank the following people for helping me make this journey or making the journey so much fun: Brian Goetz, Chris Brumme, Chris Laffra, Dawid Weiss, Erik Meijer, Jb Evain, John Rose, Mads Torgersen, Mark Reinhold, Volker Berlin, Wayne Kovsky, The GNU Classpath Community, The Mono Community.

And I want to especially thank my friend Miguel de Icaza for his guidance, support, inspiration and tireless efforts to promote IKVM.

Thank you all and goodbye.

[NOTE: This article talks about commercial products and contains links to them, I do not receive any money if you buy those tools, nor I work for or I am affiliated to any of those companies. The opinion expressed here are mine and the review is subjective]

This is my attempt at a review of Spitfire Audio BT Phobos. Before diving into the review, and since I know I will be critic particularly on some aspects, I think it’s fair to assess the plugin right away: BT Phobos is an awesome tool, make no mistakes.

BT Phobos is a “polyconvolution” synthesiser. It is, in fact, the first “standalone” plugin produced by Spitfire Audio, which is one of the companies I respect the most when it comes to music production and sample based instruments.

The term polyconvolution is used by the Spitfire Audio team to indicate the simultaneous use of three convolvers for four primary audio paths: you can send any amount of each of those four primary sources (numbered 1 to 4) outputs to each of the three convolution engines (named W, X and Y).

Screen Shot 2017-04-18 at 13.31.39

Source material controls

There is lot of flexibility in the mixing capabilities; there are, of course, separate dry/wet signal knobs that send a specific portion of the unprocessed source material to the “amplifier” module, control how much of the signal goes to the convolution circuits, and finally how much of each of the convolution engines applies to each of the source sound.

This last bit is achieved by means of an interesting nabla shaped X/Y pad: by positioning the icon that represents the source module closer to a corner it’s possible activate just the convolution engine that represents that corner; for example, top left is the W engine, top right the X and bottom the Y. Manually moving the icon gradually introduces contributions from the other engines, and double clicking on the icon makes all convolvers contribute equally to the wet sound, by positioning them to the center of the nabla.

Screen Shot 2017-04-18 at 15.45.15

The convolution mixer

Finally, each convolver has a control that allows to change the output level of the convolution engine before it reaches its envelope shaper. Spitfire Audio has released a very interesting flow diagram that shows the signal path in detail, which is linked below for reference.

BT Phobos signal path

In addition to the controls just described, the main GUI has basic controls to tweak the source material with an ADSR envelope which is directly accessible below each of the main sound sources as well as the convolutions modules, but it’s possible to have access to more advanced settings by clicking on the number or the letter that identifies the module name.

Screen Shot 2017-04-18 at 13.32.21

The advanced controls interface

An example of such controls is the Hold parameter, which let the user adjust the time the sound is held at full level before entering the Decay phase of its envelope; another useful tool is the sampling and IR offset controls, which allow to tweak parameters like the starting point of the material or the quantisation and its Speed (the playback speed for the samples, and is a function of the host tempo); there is also a control to influence the general pitch of the sound; finally a simple but effective section is dedicated to filtering – although a proper EQ is missing – as well as panning and level adjustments.

All those parameters are particularly important settings when using loops, but also contribute to shaping the sound with the pitched material, and can be randomised for interesting effects and artefacts generated from the entropy (you can just randomise the material selection only as opposed to all the parameters).

Modulation is also present, of course, with various LFOs of various kind that can be used to modulate basically everything. You can access them either by clicking on the mappings toggle below the ADSR envelope of each section, or by using the advanced settings pages.

The amount of tweaks that can be made to the material in both the source and the convolution engines is probably the most important aspect of BT Phobos, since it gives an excellent amount of freedom to create new sounds from what’s available, which is already a massive amount of content, and allows to build wildly different patches with a bit of work, but it’s definitely not straightforward and needs time to understand the combined effects that each setting has on the whole.

Since the material is polyphonic, the Impulse Responses for the convolution are created on the fly, and in fact, one interesting characteristic of BT Phobos is that there is no difference between a material for the convolution engines and one for the source module,  both draw from the same pool of sounds.

Screen Shot 2017-04-18 at 14.41.57

BT Phobos beautiful GUI

There is a difference on the type of material though, where loop based samples are, well, looped (and tempo sync’ed), and their pitch does not change based on the key that triggers them (although you can still affect the general pitch of the sound with the advanced controls), “tonal” material are pitched and change following the midi notes.

One note about the LFOs: the mappings are “per module”. In other words, it is possible to modulate almost every parameter inside a single module, be it one of the four input sources or one of the three convolution engines, but there seem to be no way to define a global mapping of some kind. For example, I found a very nice patch from Mr. Christian Henson (which incidentally made, at least in my opinion, the best and most balanced overall presets), and I noticed I could make it even more interesting by using the modulation wheel. I wanted to modulate the CC1 message with an LFO (in fact, ideally it would be even better to have access to a custom envelope, but BT Phobos doesn’t have any for modulation use), but I could not find a way to do that other than using Logic’s own Midi FX. I understand that MIDI signals are generated outside the scope of the plugin, but it would be fantastic to have the option of tweaking and modulate everything from within the synth itself.

All the sources and convolvers can be assigned to separate parts of the keyboard by tweaking the mapper at the bottom of the GUI. It is not possible to map a sound to start from an offset in the keyboard controls – for example to play C1 on the keyboard but trigger C2, or any other note – but of course you can change the global pitch so this has effectively the same result, and as said before it can also be modulated with an LFO or via DAW automation, for more interesting effects.

Screen Shot 2017-04-18 at 21.42.19

Keyboard mapping tool

Indeed, the flexibility of the tool, and the number of options at disposal for tweaking the sounds are very impressive. Most patches are very nice and ready to be used as they are, and blend nicely with lots of disparate styles. Some patches are very specific though, and pose a challenge to be used. Generally, I would consider these as starting points for exploration, rather than “final”.

When reading about BT Phobos in the weeks before its release many people asked whether you could add your own sound to it or not. It’s not possible, unfortunately.

At first, I thought that wasn’t a limitation or a deal breaker. I still think it’s not a deal breaker, but I see the value added that BT Phobos has even just as a standalone synth, as opposed to recreate the same kind of signal path manually with external tools, to give your own content the “Phobos treatment”, which is something that is entirely possible of course, for example just with Alchemy and Space Designer (which are both included in MainStage, so you can get them for a staggering 30 euros if you are a Mac user, even if you don’t use Logic Pro X!), but of course, we would be trading away the immediacy that BT Phobos delivers.

That, maybe, is my main criticism to this synth, and I hope Spitfire Audio turns BT Phobos into a fully fledged tool for sound design over time, maybe enabling access to spectral shaping in some form or another, so we can literally paint over (or paint away!) portions of the sound, which is something you can do with iZotope Iris or Alchemy and is a very powerful way to shape a sound and do sound design in general.

Another thing that is missing is a sound effect module, although I don’t know how important that is, given that there are thousands of outstanding plugins that do all sort of effects from delay to chorus etc… And, in fact, many patches benefit for added reverb (I use Eventide Blackhole and found that works extremely well with BT Phobos, since it’s also prominently used for weird sound effects). But it may be interesting to play by putting some effects (including a more proper EQ section) in various places in the signal path, although it’s all too easy to generate total chaos from such experimentation, so it’s possible the Spitfire Audio simply thought to leave this option for another time and instead focus on a better overall experience.

And there’s no arpeggiator! Really!

The number of polyphonic voices can be altered. Spitfire Audio states that the synth tweaks the number of voices at startup to match the characteristics of your computer, but I can’t confirm that, since every change I do seems to remain, even if I occasionally hear some pop and cracks at higher settings. Nevertheless, the CPU usage is pretty decent unless you go absolutely crazy with the polyphony count. I also noted that the numbers effect the clarity of the sound. This is understandable since an higher count means more notes can be generated at the same time, which means more things are competing for the same spectrum, and things can become very confusing very quickly. On the other end, a lower polyphony count has a bad impact on how the notes are generated. I feel sometime that things just stop generating sound, which is counter intuitive and very disturbing, especially since it’s very easy to have a high polyphony count with all those sources and convolvers.

Also to note is that, by nature, some patches have very wild difference in their envelopes and level settings, which means it’s all to easy to move from a quiet to a very loud patch just by clicking “next” (which is possible in Logic at least with the next/prev patch buttons on top of the plugin main frame). The synth does not stop the sound, nor does any attempt to fade from one sound to the next, instead, the convolutions simply keep working on the next sample in queue with the new settings! I still have to decide if this is cool or not, perhaps it’s not intentional, but I can see how this could be used to automate patch changes in some clever way during playback! And indeed, a was able to create a couple of interesting side effects just by changing between patches at the right time.

More on the sounds. The amount of content is really staggering, and simply cycling through the patches does not make justice to this synth, at all!

What BT Phobos wants is a user that spends time tweaking the patches and play with the source material to get the most out it, however it’s easy to see how limiting this may feel at the same time, particularly with the more esoteric and atonal sounds, and there’s certainly a limit on how good a wood stick convolved with an aluminium thin can may sound, so indeed some patches do feel repetitive at times, as the source material does. There are quite a few very similar drum loops for example, or various pitches “wind blowing into a pipe” kind of things.

This is a problem common to other synths based on the idea of tweaking sounds from the environment, though. For example, I have the amazing Geosonics from Soniccouture, which is an almost unusable library that, once tweaked, is capable of amazing awesomeness. Clearly, the authors of both synths – but this is especially valid for BT Phobos I think – are looking at an audience that is capable of listening through the detuned and dissonant sound waves and shape a new form of music.

This is probably the reason why so many of the pre assembled patches dive the user full speed into total sound design territory; however, and this is another important point of criticism, this is sound design that has already been done for you… A lot of the BT patches, in particular, are clearly BT patches, using them as they are means you are simply redoing something that has already been done before, and, despite with a very experimental feeling still strongly present, it’s not totally unheard or new.

For example, I also happen to have Break Tweaker and Stutter Edit (tools that also originally come from BT), and I could not resist to the temptation to play something that resembles BT work on “This Binary Universe” or “_” (fantastic albums)! While this seems exciting – BT in a box! And you can also see the democratising aspect of BT Phobos, I can do that in half hour instead of six months of manual CSound programming! – it’s an unfortunate and artificial limitation on a tool that is otherwise a very powerful enabler, capable of bringing complex sound design one step closer to the general public. Having the ability to process your own sounds would mitigate this aspect I think.

I do see how this is useful for a composer in need of a quick solution for an approaching deadline even with the most experimental tones, though: those patches can resolve a deadlock or take you out of an impasse in a second.

The potential for BT Phobos to become a must have tool for sound design are all there, especially if Spitfire Audio keeps adding content, perhaps more varied (and even better, allow to load your own content). The ability to shape the existing sounds already make it very usable. I don’t think it’s a general tool at this stage, though, and definitely it should not be the first synth or sound shaping processor in your arsenal, especially if you are starting out now.

But it’s not just a one trick pony either, it does offer you quite a lot of possibilities, and the more you work on that, the more addictive it becomes, and I can see Spitfire Audio offering soon this synth within a collection comprising of some of their more experimental stuff like LCO and Enigma, which would be very nice, indeed.

It’s unfortunate that Spitfire Audio does not offer an evaluation period: contrary to most of their offering, BT Phobos needs time to be fully grasped and it’s all but immediate (well, unless you are happy with the default patches or you really just need to “get out of troubles” quickly, but be careful with that because the tax is on the originality), but it can, and does, evolve, as its convolutions do, over time and it can absolutely deliver total awesomeness if used correctly.

Most patches are also usable out of the box, and especially by adding some reverb or doing some post processing with other tools, it’s possible to squeeze even more life out of them.

Overall, I do recommend BT Phobos, is a wonderful, very addictive synthesiser.


Quantum Curling

Last week we had a work week at Mozilla’s Toronto office for a bunch of different projects including Quantum DOM, Quantum Flow (performance), etc. It was great to have people from a variety of teams participate in discussions and solidify (and change!) plans for upcoming Firefox releases. There were lots of sessions going on in parallel and I wasn’t able to attend them all but some of the results were written up by the inimitable Ehsan in his fourth Quantum Flow newsletter.

Near the end of the week, Ehsan gave an impromptu walkthrough of the Gecko profiler. I’m planning on taking some of the tips he gave and that were discussed and put them onto the documentation for the profiler. If you’re interested in helping, please let me know!

The photo above is of us going curling at the High Park Curling Club. It was a lot of fun and I was happy that only one other person had ever curled before so it was a unique experience for almost everyone!

As previously reported, the JSR 269 annotation processing APIs in the javax.lang.model and javax.annotation.processing packages are undergoing maintenance review as part of Java SE 9.

All the planned changes to the JSR 269 API are in JDK 9 build 164, downloadable as an early access binary. Of note new in build 164 is the annotation type javax.annotation.processing.Generated, meant to be a drop-in replacement for javax.annotation.Generated since the latter is not in a convenient module.

Please try out your existing annotation processors -- compiling them, running them, etc. -- on JDK 9 and report your experiences, good or bad, to compiler-dev@openjdk.java.net.

As has been done previously during Java SE 7 and Java SE 8, the JSR 269 annotation processing API is undergoing a maintenance review (MR) as part of Java SE 9.

Most of the API changes are in support of adding modules to the platform, both as a language structure in javax.lang.model.* as well as another interaction point in javax.annotation.processing in the Filer and elsewhere. A small API change was also done to better support repeating annotations. A more detailed summary of the API changes is included in the MR material.

The API changes are intended to be largely compatible with the sources of existing processors, their binary linkage, as well as their runtime behavior. However, it would be helpful to verify that your existing processors work as expected when run under JDK 9. JDK 9 early access binaries are available for download. Please report experiences running processors under JDK 9 as comments here or to me as email. Feedback on the API changes can be sent to compiler-dev@openjdk.java.net.

Note: this article is also available in German.

What is Conversations?

Conversations is an app for Android Smartphones for sending each other messages, pictures, etc, much like WhatsApp. However, there are a number of important differences to WhatsApp:

  • Conversations does not use your phone number for identification, and doesn’t read your address book to find contacts. It uses an ID that looks much like an email address (the so-called Jabber-ID), and you can find contacts by exchanging Jabber-IDs with people, just like you do with email addresses, phone numbers, etc.
  • Conversations uses an open protocol called XMPP, that is used by many other programs on a wide range of systems, for example on desktop PCs.
  • Converations is Open Source, i.e. everybody can inspect the source code, check it for security issues, see what the program actually does, or even modify and distribute it.
  • XMPP builds on a decentralized infrastructure. This means that not one company is in control of it, but instead there are many providers, or you can even run your own server if you want.
  • Conversations does not collect and sell any information from you or your contacts.

There are more differences, but I don’t want to go into detail here, others have already done it, and better (German).

Install Conversations

From Google Play

Conversations is easily installed from Google Play. However, it currently costs 2,39€. I’d recommend everybody who can to buy the it, it supports development of this really good app.

Alternative: From F-Droid

For all those who cannot or don’t want to spend the money, there is another way to get it for free. It is available in the F-Droid. It is an alternative app store, that only distributes Open Source software. In order to do that, you first need to install F-Droid. Then you can start F-Droid and search for Conversations and install it.

Set-up Jabber account

Next step is to set up a Jabber account. You need two things: an ID, and a provider. The first part, the ID, you can choose freely, e.g. a fantasy name or something like firstname.surname, but this is really up to you. In order to find a provider, I recommend this list https://gultsch.de/compliance_ranked.html. The providers at the top of the list have best support for the XMPP features that are relevant for smartphone users. I’d recommend trashserver.net because this supports in-band registration (directly from Conversations) and is very well maintained. If you want to further support the developer of Conversations, I’d recommend an account on conversations.im, this currently costs 8€/year. I think it is worth it, but you have the choice.

If you choose, for example, the ID ‘joe.example’ on the provider ‘provider.org’, then your Jabber-ID is joe.example@provider.org. When you’re decided on a Jabber-ID, you can easily register an account by starting Conversations, entering the Jabber-ID in the set-up screen, check the box ‘register new account on server’, enter your preferred password 2x and confirm it.

Adding contacts

Adding contacts is different than WhatsApp. You have to manually add contacts to your roster. Tap on the ‘+’ symbol next to the little people icon, enter your contact’s Jabber-ID and confirm it. Now you’re ready to start chatting. Have fun!

Diesen Artikel gibt es auch in Englisch.

Was ist Conversations?

Conversations ist eine App für Android Smartphones, mit der man sich gegenseitig Nachrichten, Bilder, etc schicken kann, sehr ähnlich wie WhatsApp. Es gibt allerdings ein paar wichtige Unterschiede zu WhatsApp:

  • Conversations verwendet nicht Deine Telefon-Nummer zur Identifikation, und nicht Dein Adressbuch um Kontakte zu finden. Deine ID sieht aus wie eine Email-Adresse (Deine sogenannte Jabber-ID), und Kontakte findest Du indem Du Deine Jabber-ID mit Bekannten austauschst, genau wie bei Email-Adressen oder Telefonnummern auch.
  • Conversations benutzt ein offenes Protokoll, genannt XMPP, das von vielen anderen Programmen auf vielen verschiedenen Systemen genutzt werden kann, z.B. auch auf Desktop PCs.
  • Conversations ist Open Source, d.h. jeder kann den Quellcode einsehen, und z.B. auf Sicherheitsprobleme überprüfen, oder sich vergewissern was das Programm eigentlich macht, oder es ändern, etc.
  • XMPP baut auf eine dezentrale Infrastruktur, das bedeutet daß nicht ein Unternehmen alles kontrolliert, sondern daß es viele verschiedene Anbieter gibt, oder man z.B. selbst entsprechende Server betreiben kann, wenn man möchte.
  • Conversations sammelt und verkauft keinerlei Informationen über Dich und Deine Kontakte.

Es gibt noch einige andere Unterschiede, aber ich will hier nicht im Detail darauf eingehen, das haben andere schon viel besser getan.

Conversations installieren

Von Google Play

Conversations lässt sich ganz einfach von Google Play installieren. Es kostet dort allerdings momentan 2,39€. Ich möchte allen, die die Möglichkeit haben empfehlen, die App zu kaufen, ihr unterstützt damit die Entwicklung dieser wirklich guten App.

Alternative: Von F-Droid

Allen, die Google Play nicht nutzen können, oder die aus welchen Gründen auch immer nicht 2,39€ dafür zahlen können oder möchten, sei die Installation über F-Droid ans Herz gelegt. F-Droid ist ein alternativer App Store, der aussschliesslich Open Source Software bereitstellt. Dazu muss man sich zunächst F-Droid installieren. Dann kann man in der F-Droid App nach ‘Conversations’ suchen, und dort installieren. Das ist kostenlos und legal.

Jabber-Account einrichten

Als nächstes musst Du einen Jabber Account einrichten. Dazu benötigst Du zwei Dinge: eine ID, und einen Anbieter. Den ersten Teil, die ID, kannst Du selbst wählen, z.B. einen Phantasienamen, oder etwas wie vorname.nachname, aber das ist wirklich Dir überlassen. Um einen Anbieter zu finden, empfehle ich diese Liste: https://gultsch.de/compliance_ranked.html. Die Anbieter ganz oben unterstützen die meisten Features die für Smartphone-Nutzer wichtig sind. Ich kann trashserver.net empfehlen, da dieser Server die Registrierung direkt aus der Conversations-App heraus unterstützt, und auch sonst sehr gut gewartet wird. Wenn Du dem Entwickler von Conversations zusätzlich Unterstützung zukommen lassen möchtest, dann ist ein Account auf conversations.im empfehlenswert, dies kostet aber momentan 8 Euro im Jahr. Ich finde, das ist es wert, aber das muss jeder selbst entscheiden.

Wenn Du z.B. die ID ‘max.mustermann’ auf dem Anbieter ‘anbieter.de’ aussuchst, ist Deine Jabber-ID: max.mustermann@anbieter.de . Wenn Du Dich für eine ID und einen Anbieter entschieden hast, dann kannst Du ganz einfach einen Account einrichten, indem Du Conversations startest und im Einrichtungs-Bildschirm die gewünschte Jabber-ID eingibst, das Häkchen bei ‘Neues Konto auf Server erstellen’ aktivierst, Dein gewünschtest Passwort 2x eingibst und dann auf ‘Weiter’ tippst.

Kontakte hinzufügen

Anders als in WhatsApp musst Du in Conversations Deine Kontakte selbst hinzufügen. Dazu einfach auf das Symbol mit dem ‘+’ neben Männchen tippen, die Jabber-ID des Kontaktes eingeben, fertig. Und dann kannst Du loschatten! Viel Spaß!

After turning off comments on this blog a few years ago, the time has now come to remove all the posts containing links. The reason is again pretty much the same as it was when I decided to turn off the comments - I still live in Hamburg, Germany.

So, I've chosen to simply remove all the posts containing links. Unfortunately, that were pretty much all of them. I only left my old post up explaining why this blog allows no comments, now updated to remove all links, of course.

Over the past years, writing new blog posts here has become increasingly rare for me. Most of my 'social media activity' has long moved over to Twitter.

Unfortunately, I mostly use Twitter as a social bookmarking tool, saving and sharing links to things that I find interesting.

As a consequence, I've signed up for a service that automatically deletes my tweets after a short period of time. I'd link to it, but ...

This is a very important machine that really deserves to get built. Anyone who cares about Free Software should consider funding this project at some level, and spreading the word to their friends. If this project succeeds, it will bootstrap a market for new, owner-controlled performant desktop machines. If it fails, no such computers will exist. The project page and updates explain the current (rather depressing) state of general purpose computing better than I could, so take a look.

RoboVM 0.0.1 got released this week by Trillian AB.

RoboVM's main focus is to compile Java to native for deployment on mobile devices such as iOS and Android. RoboVM uses a Java to Objective-C bridge built using LLVM. Good news is that the same process work for converting Java applications to native applications on GNU/Linux systems as well!

Mario Zechner the author of libgdx posted this nice picture from inside DDD/GDB of his first HelloWorld compiled to native X86 code running on a GNU/Linux machine.
GNU/Linux machine code generated by RoboVM seen from inside DDD/GDB

http://www.robovm.org/

JogAmp is the home of high performance Java™ libraries for 3D Graphics, Multimedia and Processing.
JOGL, JOCL and JOAL provide cross platform Java™ language bindings to the OpenGL®, OpenCL™, OpenAL and OpenMAX APIs.
Running on Android, Linux, Window, OSX, and Solaris across devices using Java.

Release announcement for JogAmp 2.0.2-rc12

"You're encouraged to stop using the now-ancient 2.0-rc11!"

This 2.0.2-rc12 release include the largest security review in the 10-year history of JOGL

  • Security Fixes

    • Dynamic Linker Usage / Impl.
    • ProcAddressTable field visibility
    • Perform SecurityManager checks where required
    • Validation of property access
    • JAR Manifest tags:
      • Codebase
      • Permissions
      • Sealed
    • Use latest Java7 toolchain
      • Generating Java 1.6 bytecode
      • HTML API doc

https://jogamp.org/wiki/index.php/SW_Tracking_Report_Objectives_for_the_release_2.0.2_of_JOGL
Security fixes are marked in red on the above bug tracking page.
JogAmp send out thanks to the FuzzMyApp security researchers for healthy communication that triggered the security review work.

If you find an issue with the release, please report it to our bug database under the appropriate component. Development discussion takes place inside the JogAmp forum & mailing-list and the #jogamp IRC channel on irc.freenode.net.


Meet us @

JogAmp @ SIGGRAPH 2013

If you’ve been following Infinity and would like to, you know, download some code and try it out… well, now you can!

screenshot

The first release candidate is finally available. It can be downloaded here or from NuGet.

What's New (relative to IKVM.NET 8.0):

  • Integrated OpenJDK 8u45.
  • Many fixes to late binding support.
  • Added ikvmc support for deterministic output files.
  • Various sun.misc.Unsafe improvements.
  • Many minor bug fixes and performance tweaks.

Changes since previous development snapshot:

  • Assemblies are strong named.
  • Fix for bug #303. ikvmc internal compiler error when trying to get interfaces from type from missing assembly reference.
  • Implemented NIO atomic file move on Windows.

Binaries available here: ikvmbin-8.1.5717.0.zip

Sources: ikvmsrc-8.1.5717.0.zip, openjdk-8u45-b14-stripped.zip

Thanks to everybody who commented on the JamVM 2.0.0 release, and apologies it's taken so long to approve them - I was expecting to get an email when I had an unmoderated comment but I haven't received any.

To answer the query regarding Nashorn.  Yes, JamVM 2.0.0 can run Nashorn.  It was one of the things I tested the JSR 292 implementation against.  However, I can't say I ran any particularly large scripts with it (it's not something I have a lot of experience with).  I'd be pleased to hear any experiences (good or bad) you have.

So now 2.0.0 is out of the way I hope to do much more frequent releases.  I've just started to look at OpenJDK 9.  I was slightly dismayed to discover it wouldn't even start up (java -version), but it turned out to be not a lot of work to fix (2 evenings).  Next is the jtreg tests...

I'm pleased to announce a new release of JamVM.  JamVM 2.0.0 is the first release of JamVM with support for OpenJDK (in addition to GNU Classpath). Although IcedTea already includes JamVM with OpenJDK support, this has been based on periodic snapshots of the development tree.

JamVM 2.0.0 supports OpenJDK 6, 7 and 8 (the latest). With OpenJDK 7 and 8 this includes full support for JSR 292 (invokedynamic). JamVM 2.0.0 with OpenJDK 8 also includes full support for Lambda expressions (JSR 335), type annotations (JSR 308) and method parameter reflection.

In addition to OpenJDK support, JamVM 2.0.0 also includes many bug-fixes, performance improvements and improved compatibility (from running the OpenJDK jtreg tests).

The full release notes can be found here (changes are categorised into those affecting OpenJDK, GNU Classpath and both), and the release package can be downloaded from the file area.