Archive for consulting

« Older Entries

Catalyst, a WebAssembly-enabled version of SqueakJS

Posted in Appsterdam, consulting, Context, livecoding with tags , , , , , , , on 8 May 2023 by Craig Latta
Catalyst uses a synchronized linear representation of object memory to coordinate JavaScript and WebAssembly

It’s straightforward to apply WebAssembly (WASM) to isolated JavaScript hotspots, where there are no side-effects. This enables us to speed up sections of the SqueakJS primitives, like BitBLT, which perform pure functions. The SqueakJS interpreter code, on the other hand, is rife with side-effects. The act of interpretation modifies the deep graph of connected structures which is the Smalltalk object memory.

In my first WASM-enabled SqueakJS virtual machine, I wrote a WASM version of the JS function that interprets a single Smalltalk instruction, interpretOneWASM. To perform the necessary interactions with the object memory, that WASM function called the original JS support functions. That approach yielded a working virtual machine, but with a high performance penalty. Since JS and WASM can use shared memory, I’m curious to see if we can eliminate the JS function calls in interpretOneWASM by using a synchronized linear representation of object memory, and if this would speed up the virtual machine.

I’ve modified SqueakJS so that it maintains a shared WASM memory buffer, into which it writes and updates serializations of the objects in object memory. Now, interpretOneWASM can read and write the object information it needs without having to make most of the previous JS calls. At the moment, some primitives (like garbage collection) are still JS calls. Since they were relatively long-running operations already, I don’t expect performance to suffer much.

The format of the shared linearized object memory is nearly identical to that of an object memory snapshot, with two differences. The first is that only one object header word is written in all cases; WASM doesn’t need the additional object header information supplied in a snapshot for re-creating objects, since it doesn’t do that. The second difference is that young objects, with negative addresses, are also represented, in a special memory segment above the tenured objects. This scheme allows WASM to access all objects by address.

When WASM modifies the object memory, it records the addresses of the modified objects in a special object. The JS side uses this information to updates the canonical JS object memory representation. If at some point we implement all of the JS functions in WASM, we’ll be able to make the linear representation the only one. In the meantime, we have a framework for using both JS and WASM to their strengths, and transitioning from JS to WASM gradually.

The user can choose dynamically whether the JS or the WASM version of the interpretOne function is in use. At the moment, we let JS read the object memory snapshot and get the system started, then switch to WASM after the initial context is loaded and running. JS also writes the entire linearized object memory into the shared WASM buffer. The user can switch interpretOne from JS to WASM by evaluating a Smalltalk expression that invokes a JS function of the interpreter (“JS display vm useWASM”).

It’ll be interesting to see if this scheme yields an overall increase in system speed. Hopefully, with this gradual transition, we’ll be able to tell if a full conversion of SqueakJS from JS to WASM is worthwhile, before investing a lot of effort into rewriting the JS functions.

1 Comment »

a WebAssembly Squeak virtual machine is running

Posted in Appsterdam, Caffeine, consulting, Context, livecoding, Smalltalk, Spoon, SqueakJS with tags , , , , , , , , on 14 April 2023 by Craig Latta
the instructions are ticking!

I’ve replaced the inner instruction-dispatch loop of a running SqueakJS virtual machine with a handwritten WebAssembly (WASM) function, and run several thousand instructions of the Caffeine object memory. The WASM module doesn’t yet have its own memory. It’s using the same JavaScript objects that the old dispatch loop did, and the supporting JS state and functions (like Squeak.Interpreter class). I wrote a simple object proxy scheme, whereby WASM can use unique integer identifiers to refer to the Smalltalk objects.

Because of this indirection, the current performance is very slow. The creation of an object proxy is based on stable object pointer (OOP) values; young objects require full garbage collection to stabilize their OOPs. There is also significant overhead in calling JavaScript functions from WASM. At this stage, the performance penalties are worthwhile. We can verify that the hybrid JS/WASM interpreter is working, without having to write a full WASM implementation first.

a hybrid approach

My original approach was to recapitulate the Slang experience, by using Epigram to decompile the Smalltalk methods of a virtual machine to WASM. I realized, though, that it’s better to take advantage of the livecoding capacity of the SqueakJS VM. I can replace individual functions of the SqueakJS VM, maintaining a running system all the while. I can also switch those functions back and forth while the system is running, perhaps many millions of instructions into a Caffeine session. This will be invaluable for debugging.

The next goal is to duplicate the object memory in a WASM memory, and operate on it directly, rather than using the object proxy system. I’ll start by implementing the garbage collector, and testing that it produces correct results with an actual object memory, by comparing its behavior to that of the SqueakJS functions.

Minimal object memories will be useful in this process, because garbage collection is faster, and there is less work to do when resuming a snapshot.

performance improvement expected

From my experiment with decompiling a Smalltalk method for the Fibonacci algorithm into WASM, I saw that WASM improves the performance of send-heavy Smalltalk code by about 250 times. I was able to achieve significant speedups from the targeted use of WASM for the inner functions of BitBLT. From surveying performance comparisons between JS and WASM, I’m expecting a significant improvement for the interpreter, too.

5 Comments »

tactical Squeak speedups with WebAssembly

Posted in Appsterdam, Caffeine, consulting, Context, livecoding, Smalltalk, SqueakJS with tags , , , , , , , on 7 April 2023 by Craig Latta

With the JavaScript bridge in SqueakJS, we can utilize built-in web browser behavior and other JS frameworks from Smalltalk, just as any other JS code would. I’ve used it to build Caffeine apps using A-Frame and croquet.io. Another useful framework we can integrate is WebAssembly (WASM), a stack-oriented instruction set for writing high-performance code. I have begun to identify performance-critical code in the SqueakJS virtual machine, and replace it with WASM code. The initial results are encouraging and useful.

identifying hotspots

I’m running SqueakJS in the Chrome web browser. To identify virtual machine code that consumes the most time, I profile use cases that seem slow, using Chrome’s built-in devtools. The first use case I chose was drag-selecting a large quantity of text in a workspace.

a performance capture of drag-selecting text, indicating that rgbMulwith() is the most time-consuming inner BitBLT function

From reviewing a performance capture of this use case, we can see that rgbMulwith() is the most time-consuming inner function from the BitBLT plugin. While it doesn’t modify variables in outer scopes, it does read a plugin-global variable. Most of the work it does, however, is done by partitionedMulwithnBitsnPartitions(), a pure function returning the result of a mathematical operation on the inputs, without any other system state interaction. That makes it well-suited to WASM implementation.

While there are APIs for coordinating side effects with JavaScript, they are relatively slow. It is therefore harder to rationalize WASM implementations of individual higher-level Squeak virtual machine primitives, since they interact extensively with complex JavaScript objects like the Squeak interpreter. Eventually, we’ll represent the entire Squeak object memory inside a WASM memory, and implement the entire Squeak virtual machine with WASM functions. WASM garbage collection will assist the Squeak garbage collector, much as the JavaScript garbage collector assists the SqueakJS VM now. JavaScript interaction will be limited to the WASM implementation of the SqueakJS JS bridge.

translating from JS to WASM

Here’s the existing JS implementation of partitionedMulwithnBitsnPartitions():

With its stack-based instructions, WASM code is reminiscent of Smalltalk bytecode. Here’s some of the equivalent WASM implementation of the above function, written by hand. The WASM memory holds the maskTable from BitBitPlugin.js.

a section of the equivalent WASM

Note that WASM’s shift-left and shift-right instructions are fine as is; we don’t need to make wrapper functions for them as we did in JS.

After I modified the BitBLT plugin so that rgbMulwith() uses partitionedMUL(), drag-selecting text in the Caffeine user interface was much more responsive, and a different inner BitBLT plugin function was the most time-consuming. Even though rgbMulwith() used a small percentage of total time in the first performance capture, every saved millisecond significantly improves perceived animation smoothness. By using additional use cases (scrolling long lists, and repainting by alternating the stacking order of two windows), I identified other inner BitBLT plugin functions to optimize. The Caffeine user interface is now much more responsive than it was. This is especially useful with Worldly, the spatial IDE I’m building with Caffeine and A-Frame, where every bit of performance matters.

an alternative to writing WASM by hand

For the JS code in the Squeak virtual machine, it makes sense to write replacement WASM code by hand. Since WASM code is so similar to Smalltalk bytecode, for Smalltalk compiled methods it makes more sense to use automated decompilation to WASM. I have done this for a small proof-of-concept, using a Smalltalk compiled method for the Fibonacci algorithm.

Using the Smalltalk compiler and decompiler I wrote with my Epigram parsing framework, I was able to decompile the Smalltalk compiled method for the Fibonacci algorithm into WASM text. I then used an in-browser version of the WebAssembly Binary Toolkit from Caffeine to generate binary WASM, compile it in the current page as a function, and call the function. Comparing the execution time of finding the 29th Fibonacci number in both Smalltalk and WASM showed that WASM had 250 times the execution speed of the normal SqueakJS bytecode-to-JS translator.

I plan to write, in Smalltalk, a version of the Squeak virtual machine simulator that stores all objects in a WASM memory. Once it can evaluate (3 + 4), I’ll translate all its Smalltalk compiled methods to WASM, and see how much faster it runs. The next step will be to get a JS bridge working, and implement interfaces to the web browser DOM for graphics and user input event handling. Ultimately, a WASM implementation of the Squeak virtual machine may be preferable to the SqueakJS virtual machine.

2 Comments »

realtime vocal harmonization with Caffeine

Posted in Uncategorized with tags , , , , , , , , on 14 July 2021 by Craig Latta

I’ve written a Caffeine class which, in real time, takes detected pitches from a melody and chords, and sends re-voiced versions of the chords to a harmonizer, which renders them using shifted copies of the melody. It’s an example of an aggregate audio plugin, which builds a new feature from other plugins running in Ableton Live.

re-creating a classic

Way way back in 1991, before the Auto-Tune algorithm popularized in 1998, a Canadian company called IVL Technologies developed a hardware harmonizer, the Vocalist VHM5. It generated five-part vocal harmonies, live from sung melodies and chords played via MIDI. It had a simple but effective model of vocal formants, which enabled it to shift the pitch of a sung note to natural-sounding new pitches, including correcting the pitch of the sung note. It also had very fast pitch detection.

My favorite feature, though, was how it combined those features when voicing chords. In what was called “vocoder mode”, it would adjust the pitches of incoming MIDI chords to be as close as possible to the current pitch of a sung melody, or closed voicing. If the melody moved more than half an octave away from a chord voice, the rendered chord voice would adjust by some number of octaves up or down, so as to be within half an octave of the melody. With kinetic melodies and dense chords, this becomes a simple but compelling voice-leading technique. It’s even more compelling when the voices are spatialized in a stereo or 3D audio field, with reverb, reflections, and other post-processing.

It’s also computationally inexpensive. The IVL pitch-detection and shifting algorithms were straightforward for off-the-shelf digital signal processing chips to perform, and the Auto-Tune algorithm is orders of magnitude cheaper. One of the audio plugins I use in the Ableton Live audio environment, Harmony Engine by Antares, implements Auto-Tune’s pitch shifting. Another, MIDI Guitar by Jam Origin, does polyphonic pitch detection. With these plugins, I have all the live MIDI information necessary to implement closed re-voicing, and the pitch shifting for rendering it. I suppose I would call this “automated closed-voice harmonization”.

implementation

Caffeine runs in a web browser, which, along with Live, has access to all the MIDI interfaces provided by the host operating system. Using the WebMIDI API, I can receive and schedule MIDI events in Smalltalk, exchanging music information with Live and its plugins. With MIDI as one possible transport layer, I’ve developed a Smalltalk model of music events based upon sequences and simultaneities. One kind of simultaneity is the chord, a collection of notes sounded at the same time. In my implementation, a chord performs its own re-voicing, while also taking care to send a minimum of MIDI messages to Live. For example, only the notes which were adjusted in response to a melodic change are rescheduled. The other notes simply remain on, requiring no sent messages. Caffeine also knows how many pitch-shifted copies of the melody can be created by the pitch-shifting plugin, and culls the least-recently-activated voices from chords, to remain within that number.

All told, I now have a perfect re-creation of the original Vocalist closed-voicing sound, enhanced by all the audio post-processing that Live can do.

the setup

a GK-3 hex pickup through a breakout box

Back in the day, I played chords to the VHM5 from an exotic MIDI electric guitar controller, the Zeta Mirror 6. This guitar has a hex (six-channel) pickup, and can send a separate data stream for each string. While I still have that guitar, I also have a Roland GK-3 hex pickup, which is still in production and can be moved between guitars without modifying them. Another thing I like about hex pickups is having access to the original analog signal for each string. These days I run the GK-3 through a SynQuaNon breakout module, which makes the signals available at modular levels. The main benefit of this is that I can connect the analog signals directly to my audio interface, without software drivers that may become unsupported. I have a USB GK-3 interface, but the manufacturer never updated the original 32-bit driver for it.

Contemporary computers can do polyphonic pitch detection on any audio stream, without the use of special controller hardware. While the resulting MIDI stream uses only a single channel, with no distinction between strings, it’s very convenient. The Jam Origin plugin is my favorite way to produce a polyphonic chord stream from audio.

the ROLI Lightpad

My favorite new controller for generating multi-channel chord streams is the ROLI Lightpad. It’s a MIDI Polyphonic Expression (MPE) device, using an entire 16-channel MIDI port for each instrument, and a separate MIDI channel for each note. This enables very expressive use of MIDI channel messages for representing the way a note changes after it starts. The Lightpad sends messages that track the velocity with which each finger strikes the surface, how it moves in X, Y, and Z while on the surface, and the velocity with which it leaves the surface. The surface is also a display; I use it as a five-by-five grid, which presents musical intervals in a way I find much more accessible than that of a traditional piano keyboard. There are several MPE instruments that use this grid, including the Linnstrument and the GeoShred iPad app. The Lightpad is also very portable, and modular; many of them can be connected together magnetically.

The main advantage of using MPE for vocal harmonization is associating various audio processing state with each chord voice’s separate channel. For example, the bass voice of a chord progression can have its own spatialization and equalization settings.

My chord signal path starts with an instrument, a hex or normal guitar or Lightpad. Audio and MIDI data goes from the instrument, through a host operating system MIDI interface, through Live where I can detect pitches and record, through another MIDI interface to Caffeine in a web browser, then back to Live and the pitch-shifting plugin. My melody signal path starts with a vocal performance using a microphone, through Live and pitch detection, then through pitch shifting as controlled by the chords.

Let’s Play!

Between this vocal harmonization, control of the Ableton Live API, and the Beatshifting protocol, there is great potential for communal livecoded music performance. If you’re a livecoder interested in music, I’d love to hear from you!

Leave a comment »

a tour of Caffeine

Posted in Appsterdam, consulting, Context, Smalltalk, Spoon, SqueakJS with tags , , , , , , , , , , , , , on 27 August 2018 by Craig Latta

https://player.vimeo.com/video/286872152

Here’s a tour of the slides from a Caffeine talk I’m going to give at ESUG 2018. I hope to see you there!

Leave a comment »

Livecoding other tabs with the Chrome Remote Debugging Protocol

Posted in Appsterdam, consulting, Context, Smalltalk, SqueakJS with tags , , , , , , , on 24 July 2017 by Craig Latta

Chrome Debugging Protocol

We’ve seen how to use Caffeine to livecode the webpage in which we’re running. With its support for the Chrome Remote Debugging Protocol (CRDP), we can also use it to livecode every other page loaded in the web browser.

Some Help From the Inside

To make this work, we need to coordinate with the Chrome runtime engine. For CRDP, there are two ways of doing this. One is to communicate using a WebSocket connection; I wrote about this last year. This is useful when the CRDP client and target pages are running in two different web browsers (possibly on two different machines), but with the downside of starting the target web browser in a special way (so that it starts a conventional webserver).

The other way, possible when both the CRDP client and target pages are in the same web browser, is to use a Chrome extension. The extension can communicate with the client page over an internal port object, created by the chrome.runtime API, and expose the CRDP APIs. The web browser need not be started in a special way, it just needs to have the extension installed. I’ve published a Caffeine Helper extension, available on the Chrome Web Store. Once installed, the extension coordinates communication between Caffeine and the CRDP.

Attaching to a Tab

In Caffeine, we create a connection to the extension by creating an instance of CaffeineExtension:

CaffeineExtension new inspect

As far as Chrome is concerned, Caffeine is now a debugger, just like the built-in DevTools. (In fact, the DevTools do what they do by using the very same CRDP APIs; they’re just another JavaScript application, like Caffeine is.) Let’s open a webpage in another tab, for us to manipulate. The Google homepage makes for a familiar example. We can attach to it, from the inspector we just opened, by evaluating:

self attachToTabWithTitle: 'Google'

Changing Feelings

Now let’s change something on the page. We’ll change the text of the “I’m Feeling Lucky” button. We can get a reference to it with:

tabs onlyOne find: 'Feeling'

When we attached to the tab, the tabs instance variable of our CaffeineExtension object got an instance of ChromeTab added to it. ChromeTabs provide a unified message interface to all the CRDP APIs, also known as domains. The DOM domain has a search function, which we can use to find the “I’m Feeling Lucky” button. The CaffeineExtension>>find: method which uses that function answers a collection of search results objects. Each search result object is a proxy for a JavaScript DOM object in the Google page, an instance of the ChromeRemoteObject class.

In the picture above, you can see an inspector on a ChromeRemoteObject corresponding to the “I’m Feeling Lucky” button, an HTMLInputElement DOM object. Like the JSObjectProxies we use to communicate with JavaScript objects in our own page, ChromeRemoteObjects support normal Smalltalk messaging, making the JavaScript DOM objects in our attached page seem like local Smalltalk objects. We only need to know which messages to send. In this case, we send the messages of HTMLInputElement.

As with the JavaScript objects of our own page, instead of having to look up external documentation for messages, we can use subclasses of JSObject to document them. In this case, we can use an instance of the JSObject subclass HTMLInputElement. Its proxy instance variable will be our ChromeRemoteObject instead of a JSObjectProxy.

For the first message to our remote HTMLInputElement, we’ll change the button label text, by changing the element’s value property:

self at: #value put: 'I''m Feeling Happy'

The Potential for Dynamic Web Development

The change we made happens immediately, just as if we had done it from the Chrome DevTools console. We’re taking advantage of JavaScript’s inherent livecoding nature, from an environment which can be much more comfortable and powerful than DevTools. The form of web applications need not be static files, although that’s a convenient intermediate form for webservers to deliver. With generalized messaging connectivity to the DOM of every page in a web browser, and with other web browsers, we have a far more powerful editing medium. Web applications are dynamic media when people are using them, and they can be that way when we develop them, too.

What shall we do next?

 

Leave a comment »

browser-to-browser websocket tunnels with Caffeine and livecoded NodeJS

Posted in Appsterdam, consulting, Context, Smalltalk, SqueakJS with tags , , , , , , , , on 4 July 2017 by Craig Latta

network

In our previous look at livecoding NodeJS from Caffeine, we implemented tweetcoding. Now let’s try another exercise, creating WebSockets that tunnel between web browsers. This gives us a very simple version of peer-to-peer networking, similar to WebRTC.

Once again we’ll start with Caffeine running in a web browser, and a NodeJS server running the node-livecode package. Our approach will be to use the NodeJS server as a relay. Web browsers that want to establish a publicly-available server can register there, and browser that want to use such a server can connect there. We’ll implement the following node-livecode instructions:

  • initialize, to initialize the structures we’ll need for the other instructions
  • create server credential, which creates a credential that a server browser can use to register a WebSocket as a server
  • install server, which registers a WebSocket as a server
  • connect to server, which a client browser can use to connect to a registered server
  • forward to client, which forwards data from a server to a client
  • forward to server, which forwards data from a client to a server

In Smalltalk, we’ll make a subclass of NodeJSLivecodingClient called NodeJSTunnelingClient, and give it an overriding implementation of configureServerAt:withCredential:, for injecting new instructions into our NodeJS server:

configureServerAt: url withCredential: credential
  "Add JavaScript functions as protocol instructions to the
node-livecoding server at url, using the given credential."

  ^(super configureServerAt: url withCredential: credential)
    addInstruction: 'initialize'
    from: '
      function () {
        global.servers = []
        global.clients = []
        global.serverCredentials = []
        global.delimiter = ''', Delimiter, '''
        return ''initialized tunnel relay''}';
    invoke: 'initialize';
    addInstruction: 'create server credential'
    from: '
      function () {
        var credential = Math.floor(Math.random() * 10000)
        serverCredentials.push(credential)
        this.send((serverCredentials.length - 1) + '' '' + credential)
        return ''created server credential''}';
    addInstruction: 'install server'
    from: '
      function (serverID, credential) {
        if (serverCredentials[serverID] == credential) {
          servers[serverID] = this
          this.send(''1'')
          return ''installed server''}
      else {
        debugger;
        this.send(''0'')
        return ''bad credential''}}';
    addInstruction: 'connect to server'
    from: '
      function (serverID, port, req) {
        if (servers[serverID]) {
          clients.push(this)
          servers[serverID].send(''connected:atPort:for: '' + (clients.length - 1) + delimiter + port + delimiter + req.connection.remoteAddress.toString())
          this.send(''1'')
          return ''connected client''}
        else {
          this.send(''0'')
          return ''server not connected''}}';
    addInstruction: 'forward to client'
    from: '
      function (channel, data) {
        if (clients[channel]) {
          clients[channel].send(''from:data: '' + servers.indexOf(this) + delimiter + data)
          this.send(''1'')
          return ''sent data to client''}
        else {
          this.send(''0'')
          return ''no such client channel''}}';
    addInstruction: 'forward to server'
    from: '
      function (channel, data) {
        if (servers[channel]) {
          servers[channel].send(''from:data: '' + clients.indexOf(this) + delimiter + data)
          this.send(''1'')
          return (''sent data to server'')}
        else {
          this.send(''0'')
          return ''no such server channel''}}'

We’ll send that message immediately, configuring our NodeJS server:

NodeJSTunnelingClient
  configureServerAt: 'wss://yourserver:8087'
  withCredential: 'shared secret';
  closeConfigurator

On the NodeJS console, we see the following messages:

server: received command 'add instruction'
server: adding instruction 'initialize'
server: received command 'initialize'
server: evaluating added instruction 'initialize'
server: initialized tunnel relay
server: received command 'add instruction'
server: adding instruction 'create server credential'
server: received command 'add instruction'
server: adding instruction 'install server'
server: received command 'add instruction'
server: adding instruction 'connect to server'
server: received command 'add instruction'
server: adding instruction 'forward to client'
server: received command 'add instruction'
server: adding instruction 'forward to server'

Now our NodeJS server is a tunneling relay, and we can connect servers and clients through it. We’ll make a new ForwardingWebSocket class hierarchy:

Object
  ForwardingWebSocket
    ForwardingClientWebSocket
    ForwardingServerWebSocket

Instances of ForwardingClientWebSocket and ForwardingServerWebSocket use a NodeJSTunnelingClient to invoke our tunneling instructions.

We create a new ForwardingServerWebSocket with newThrough:, which requests new server credentials from the tunneling relay, and uses them to install a new server. Another new class, PeerToPeerWebSocket, provides the public message interface for the framework. There are two instantiation messages:

  • toPort:atServerWithID:throughURL: creates an outgoing client that uses a ForwardingClientWebSocket to connect to a server and exchange data
  • throughChannel:of: creates an incoming client that uses a ForwardingServerWebSocket to exchange data with a remote outgoing client.

Incoming clients are used by ForwardingServerWebSockets to represent their incoming connections. Each ForwardingServerWebSocket can provide services over a range of ports, as a normal IP server would. To connect, a client needs the websocket URL of the tunneling relay, a port, and the server ID assigned by the relay.

As usual, you can examine and try out this code by clearing your browser’s caches for caffeine.js.org (including IndexedDB), and visiting https://caffeine.js.org/. With browsers able to communicate directly, there are many interesting things we can build, including games, chat applications, and team development tools. What would you like to build?

3 Comments »

retrofitting Squeak Morphic for the web

Posted in Appsterdam, consulting, Context, Smalltalk, Spoon, SqueakJS with tags , , , , , , , , on 30 June 2017 by Craig Latta

Google ChromeScreenSnapz022

Last time, we explored a way to improve SqueakJS UI responsiveness by replacing Squeak Morphic entirely, with morphic.js. Now let’s look at a technique that reuses all the Squeak Morphic code we already have.

many worlds, many canvases

Traditionally, Squeak Morphic has a single “world” where morphs draw themselves. To be a coherent GUI, Morphic must provide all the top-level effects we’ve come to expect, like dragging windows and redrawing them in their new positions, and redrawing occluded windows when they are brought to the top. Today, this comes at an acceptable but noticeable cost. Until WebAssembly changes the equation again, we want to do all we can to shift UI work from Squeak Morphic to the HTML5 environment hosting it. This will also make the experience of using SqueakJS components more consistent with that of the other elements on the page.

Just as we created an HTML5 canvas for morphic.js to use in the last post, we can do so for individual morphs. This means we’ll need a new Canvas subclass, called HTML5FormCanvas:

Object
  ...
    Canvas
       FormCanvas
         HTML5FormCanvas

An HTML5FormCanvas draws onto a Form, as instances of its parent class do, but instead of flushing damage rectangle from the Form onto the Display, it flushes them to an HTML5 canvas. This is enabled by a primitive I added to the SqueakJS virtual machine, which reuses the normal canvas drawing code path.

Accompanying HTML5FormCanvas are new subclasses of PasteUpMorph and WorldState:

Object
  Morph
    ...
      PasteUpMorph
        HTML5PasteUpMorph

Object
  WorldState
    HTML5WorldState

HTML5PasteUpMorph provides a message interface for other Smalltalk objects to create HTML5 worlds, and access the HMTL5FormCanvas of each world and the underlying HTML5 canvas DOM element. An HTML5WorldState works on behalf of an HTML5PasteUpMorph, to establish event handlers for the HTML5 canvas (such as for keyboard and mouse events).

HTML5 Morphic in action

You don’t need to know all of that just to create an HTML5 Morphic world. You only need to know about HTML5PasteUpMorph. In particular, (HTML5PasteUpMorph class)>>newWorld. All of the traditional Squeak Morphic tools can use HTML5PasteUpMorph as a drop-in replacement for the usual PasteUpMorph class.

There are two examples of single-window Morphic worlds in the current Caffeine release, for a workspace and classes browser. I consider these two tools to be the “hello world” exercise for UI framework experimentation, since you can use them to implement all the other tools.

We get an immediate benefit from the web browser handling window movement and clipping for us, with opaque window moves rendering at 60+ frames per second. We can also interleave Squeak Morphic windows with other DOM elements on the page, which enables a more natural workflow when creating hybrid webpages. We can also style our Squeak Morphic windows with CSS, as we would any other DOM element, since as far as the web browser is concerned they are just HTML5 canvases. This makes effects like the rounded corners and window buttons trays that Caffeine uses very easy.

Now, we have flexible access to the traditional Morphic tools while we progress with adapting them to new worlds like morphic.js. What shall we build next?

Leave a comment »

Pharo comes to Caffeine and SqueakJS

Posted in Appsterdam, consulting, Context, GLASS, Naiad, Seaside, Smalltalk, Spoon, SqueakJS with tags , , , , , , , , , on 29 June 2017 by Craig Latta

Google ChromeScreenSnapz025

The Caffeine web livecoding project has added Pharo to the list of Smalltalk distributions it runs with SqueakJS. Bert Freudenberg and I spent some time getting SqueakJS to run Pharo at ESUG 2016 in Prague last summer, and it mostly worked. I think Bert got a lot further since then, because now there are just a few Pharo primitives that need implementing. All I’ve had to do so far this time is a minor fix to the input event loop and add the JavaScript bridge. The bridge now works from Pharo, and it’s the first time I’ve seen that.

Next steps include getting the Tether remote messaging protocol and Snowglobe app streaming working between Pharo and Squeak, all running in SqueakJS. Of course, I’d like to see fluid code-sharing of all kinds between Squeak, Pharo, and all the other Smalltalk implementations.

So, let the bugfixing begin! :)  You can run it at https://caffeine.js.org/pharo/. Please do get in touch if you find and fix things. Thanks!

2 Comments »

a faster Morphic with morphic.js

Posted in Appsterdam, consulting, Context, Naiad, Smalltalk, Spoon, SqueakJS with tags , , , , , , , , , , on 28 June 2017 by Craig Latta

Google ChromeScreenSnapz017

Caffeine is powered by SqueakJS. The performance of SqueakJS is amazingly good, thanks in large part to its dynamic translation of Smalltalk compiled methods to JavaScript functions (which are in turn translated to machine code by your web browser’s JS engine). In the HTML5 environment where SqueakJS finds itself, there are several other tactics we can use to further improve user interface performance.

Delegate!

In a useful twist of fate, SqueakJS emerges into a GUI ecosystem descended from Smalltalk, now brimming with JavaScript frameworks to which SqueakJS can delegate much of its work. To make Caffeine an attractive environment for live exploration, I’m addressing each distraction I see.

The most prominent one is user interface responsiveness. SqueakJS is quite usable, even with large object memories, but its Morphic UI hasn’t reached the level of snappiness that we expect from today’s web apps. Squeak is a virtual machine, cranking away to support what is essentially an entire operating system, with a process scheduler, window system, compiler, and many other facilities. Since, with SqueakJS, that OS has access to a multitude of similar behavior in the JavaScript world, we should take advantage.

Of course, the UI design goals of the web are different than those of other operating systems. Today’s web apps are still firmly rooted in the web’s original “page” metaphor. “Single Page Applications” that scroll down for meters are the norm. While there are many frameworks for building SPAs, support for open-ended GUIs is uncommon. There are a few, though; one very good one is morphic.js.

morphic.js

Morphic.js is the work of Jens Mönig, and part of the Snap! project at UC Berkeley, a Scratch-like environment which teaches advanced computer science concepts. It’s a standalone JavaScript implementation of the Morphic UI framework. By using morphic.js, Squeak can save its cycles for other things, interacting with it only when necessary.

To use morphic.js in Caffeine, we need to give morphic.js an HTML5 canvas for drawing. The Webpage class can create new DOM elements, and use jQuery UI to give them effects like dragging and rotation. With one line we create a draggable canvas with window decorations:

canvas := Webpage createWindowOfKind: 'MorphicJS'

Now, after loading morphic.js, we can create a morphic.js WorldMorph object that uses the canvas:

world := (JS top at: #WorldMorph) newWithParameters: {canvas. false}

Finally, we need to create a rendering loop that regularly gets the world to draw itself on the canvas:

(JS top)
  at: #world
  put: world;
  at: #morphicJSRenderingLoop
  put: (
    (JS Function) new: '
      requestAnimationFrame(morphicJSRenderingLoop)
      world.doOneCycle()').

JS top morphicJSRenderingLoop

Now we have an empty morphic.js world to play with. The first thing to know about morphic.js is that you can get a world menu by control-clicking:

Google ChromeScreenSnapz018

Things are a lot more interesting if you choose development mode:

Google ChromeScreenSnapz019.png

Take some time to play around with the world menu, creating some morphs and modifying them. Note that you can also control-click on morphs to get morph-specific menus, and that you can inspect any morph.

Google ChromeScreenSnapz020.png

Also notice that this user interface is noticeably snappier than the current SqueakJS Morphic. MorphicJS isn’t trying to do all of the OS-level stuff that Squeak does, it’s just animating morphs, using a rendering loop that is runs as machine code in your web browser’s JavaScript engine.

Smalltalk tools in another world, with Hex

The inspector gives us an example of a useful morphic.js tool. Since we can pass Smalltalk blocks to JavaScript as callback functions, we have two-way communication between Smalltalk and JavaScript, and we can build morphic.js tools that mimic the traditional Squeak tools.

I’ve built two such tools so far, a workspace and a classes browser. You can try them out with these expressions:

HexMorphicJSWorkspace open.
HexMorphicJSClassesBrowser open

“Hex” refers to a user interface framework I wrote called Hex, which aggregates several JavaScript UI frameworks. HexMorphicJSWorkspace and HexMorphicJSClassesBrowser are subclasses of HexMorphicJSWindow. Each instance of every subclass of HexMorphicJSWindow can be used either as a standalone morphic.js window, or as a component in a more complex window. This is the case with these first two tools; a HexMorphicJSClassesBrowser uses a HexMorphicJSWorkspace as a pane for live code evaluation, and you can also use a HexMorphicJSWorkspace by itself as a workspace.

With a small amount of work, we get much snappier versions of the traditional Smalltalk tools. When using them, SqueakJS only has to do work when the tools request information from them. For example, when a workspace wants to print the result of evaluating some Smalltalk code, it asks SqueakJS to compile and evaluate it.

coming up…

It would be a shame not to reuse all the UI construction effort that went into the original Squeak Morphic tools, though. What if we were to put each Morphic window onto its own canvas, so that SqueakJS didn’t have to support moving windows, clipping and so on? Perhaps just doing that would yield a performance improvement. I’ll write about that next time.

1 Comment »

« Older Entries