Greetings… I’m accepting new clients for my consultancy, Black Page Digital. I have extensive experience with several dynamic languages, virtual machine development, web services, bridging between languages and to static systems, GUI development, and live debugging. Please get in touch; I’d love to speak with you about your development efforts. Thanks!
App streaming is a way of delivering the user experience of an app without actually running the app on the user’s machine. The name is an allusion to music and video streaming; you get to experience the asset immediately, without waiting for it to download completely. Streaming an app also has the benefit of avoiding installation, something which can be problematic to do (and to undo). This is nice when you just want to demo an app, before deciding to install it.
Another advantage of app streaming is that the app can run on a much faster machine than the user’s, or even on a network of machines. Social networks are a crude example of app streaming; there are massive backends working behind your web browser, crunching away on all that graph data. Typically, though, app streaming involves an explicit visual component, with the user’s display and input devices standing in for the normal ones. The goal is to make using a new app as simple as playing an online video.
distributing Smalltalk user interface components
Everything in Smalltalk happens by objects sending messages to each other. With a remote messaging framework like Tether, we can put some of the objects in a user interface on a remote machine. Snowglobe is an adaptation of Squeak‘s Morphic user interface framework which runs Squeak on a server, but uses SqueakJS in a client web browser as the display. This is an easy way to recast a Smalltalk application as a web app, while retaining the processing speed and host platform access of the original.
Morphic is built around a display loop, where drawable components (morphs) are “stepped” at some frequency, like a flipbook animation. Normally, drawing is done on a single morph that corresponds to the display of the machine where Squeak is running. Snowglobe adds a second display morph which is Tether-aware. When drawing to this tethered display morph, the app server translates every display operation into a compact remote message.
To maximize speed, Morphic already tries to do its drawing with as few operations as possible (e.g., avoiding unnecessary redrawing). This is especially important when display operations become remote, since network transmission is orders of magnitude slower than local drawing. Since the tethered display morph also lives in a Smalltalk object memory, we can optimize drawing operations involving graphics that are known to both sides of the connection. For example, when changing the mouse cursor to a resize icon when hovering over the corner of a window, there’s no need to send the icon over the wire, since the displaying system already has it. Instead, we can send a much smaller message requesting that the icon be shown.
For full interaction, we also need to handle user input events going back the other way. Snowglobe co-opts Morphic’s user input handling as well. With user input and display forwarded appropriately together, we achieve the seamless illusion that our app is running locally, either as a single morph amongst other local morphs, or using the entire screen.
going beyond screen-sharing
Protocols like VNC do the remote display and user input handling we’ve discussed, although they are typically more complicated to start than clicking a link in a web browser. But since both systems in a Snowglobe session are Smalltalk, we can go beyond simple screen sharing. We can use Tether to send any remote messages we want, so either side can modify the app-streaming behavior in response to user actions. For example, the user might decide to go full-screen in the web browser displaying the app, prompting SqueakJS to notify the remote app, which could change the way the app displays itself.
try it for yourself
I’ve set up an AWS server running the Squeak IDE, reachable from SqueakJS in your web browser. Be gentle… there’s only one instance running (actually two, one in Europe and one in North America, chosen for you automatically by Amazon). Please check it out and let me know what you think!
In my previous post, I introduced a new topology for distributed computation with Smalltalk: an object memory in SqueakJS in a web browser, paired by remote-messaging connection with another object memory in Cog in a native app, and connected with other SqueakJS/Cog pairs on other physical machines. The remote-messaging protocol that the memories speak is called Tether. I’ll go into a few details here.
passive frame-based messaging with WebSockets
WebSockets use callback functions to deliver messages, or frames of bytes, between conversants. The Tether protocol imposes a structure on those bytes, which are processed on each side of the connection by instances of class Tether. The first four bytes are a 32-bit tag which indicates the Smalltalk class which should interpret the rest of the frame. In the case of a remote message, this is class Tether. Successive bytes indicate the message selector to perform, the receiver of the message, and the message parameters.
The message receiver is expressed as a 32-bit key into a table of exposed objects, maintained by the Tether instance handling the connection. The Tether instances themselves expose their identities to each other at the beginning of the conversation. Objects that aren’t specified by reference to an exposed-object table (such as message selectors) are expressed through serialization.
The fact that both sides of the conversation are objects in live Smalltalk systems affords many optimizations that aren’t possible when serializing objects to a static file. For this reason, I call this live serialization. For example, when transferring a compiled method between systems, if the method’s literals are objects which already exist in the receiving system, we may write references to them rather than serializing them.
We can also take special measures when the receiving system is missing the classes whose instances we want to transfer. Instead of assuming in advance that the receiving system lacks the classes whose instances we’re transferring, and including them in our payload, as a static serialization file would, we can transfer such classes only on demand. This yields much higher accuracy in object transfer, and far fewer bytes sent over the wire. Since live serialization is part of a complete remote messaging protocol, any messages at all can be sent from either side to complete an object transfer.
With a receiver, selector, and parameters specified, the receiving system can perform the message sent from the sending system. Each object in the system is responsible for serializing itself over a Tether. If the answer to the remote message is a literal object, like a symbol or integer, it will write the bytes of its value on the Tether instance handling the message. If the answer isn’t a literal object, by default it will write a reference to itself. Developers can choose to pass objects by value or by reference as they see fit.
Tether performs every remote message in a distinct process, so that no system blocks waiting for an answer to be sent back over the network. Each remote message-send is assigned a unique identifier, and each answer is sent with the ID of its message-send as metadata, so that it can be delivered to the correct waiting process.
This scheme extends the traditional imperative messaging semantics of Smalltalk to any number of machines, and each message-send may involve receiver and parameter objects which are all on different machines. Every message-send may invoke any number of further remote messages before answering.
An object which represents an object on a remote system is called a proxy. Ideally, it forwards every message sent to it to the remote object, and so provides the illusion of transparent remote messaging. Remote messaging in Smalltalk is often done by using a proxy class which inherits and implements as few messages as possible, and overriding the handler message sent by the virtual machine when a message is not understood. This provides enough coverage to do many useful things, but some messages handled specially by the virtual machine are not forwarded. Some use cases, like remote debugging, require forwarding even those special messages.
To achieve total forwarding coverage, we must modify the virtual machine. There are some situations where this is undesirable (e.g., a lack of tools or expertise, or a requirement to use a past virtual machine unmodified). Tether uses the “does not understand” tactic above in these situations, but provides a modified virtual machine for the rest. In this virtual machine, message forwarding is triggered during method lookup for instances of a specific proxy class (which can be located anywhere in the class hierarchy). Method caching and methods implemented directly as virtual machine instructions are also appropriately adapted. There are a few messages which proxies must understand locally, to participate in live serialization. These messages are also handled specially by the virtual machine.
see for yourself
Tether is an integral part of the Context project from Black Page Digital. A demo of remote messaging between SqueakJS and Cog is available. In tomorrow’s post, I’ll discuss an everyday application of remote messaging.
Smalltalk is an image-based system. In addition to modeling an idealized processor, with its own instruction set, the Smalltalk virtual machine models the processor’s memory. The virtual machine can make a snapshot of this memory, as a normal host platform file called an image. This snapshot captures the complete execution state of the Smalltalk system (the object memory), including the program counters of every process that was running at the time. The virtual machine can also resume this snapshot, restoring that system state so that the system may continue. This is similar to the “sleep” function of a laptop.
Since the virtual machine provides a consistent platform abstraction above whatever host is running it, we can suspend the system on one host and resume it on another. Smalltalk programmers have taken great advantage of this for years, writing single systems which run on multiple operating systems (e.g., Macintosh and Windows). Java attempted to provide consistent execution semantics, under the tagline “write once, run anywhere” (with mixed results), but this did not extend to a continuous memory image. We can make an object memory snapshot with SqueakJS in a web browser, and resume it with Cog in a native app.
SqueakJS uses the HTML5 “indexed database” feature for persistent storage, making it look like a normal filesystem. We can write a snapshot of the running SqueakJS system this way. We can then use ThisWebpage to download the snapshot to the user’s machine. We add a link to the document in which SqueakJS runs, and synthesize a click on it, invoking the download.
We also use ThisWebpage to detect the user’s host operating system, and download a platform-specific Cog installer. I’ve written installers for macOS, Windows, and Linux. Each one downloads the Cog virtual machine, installs a platform handler for “squeak://” URLs that runs Cog, and invokes a squeak:// URL that runs our snapshot. This URL encodes the name of the snapshot file, Cog parameters (such as whether or not to run the system headless), and a base64-encoded JSON dictionary of other parameters that the Smalltalk object memory can process.
Now our object memory is running both in the browser in SqueakJS, and in Cog as a native app. The current Black Page Digital object memory for Squeak connects the two with a remote-messaging network service. When the SqueakJS-hosted instance invokes the local one, it also forks a process that periodically attempts a remote-messaging connection. When the memory resumes on a non-web host, it starts a WebSocket-based server to provide remote messaging service.
With the two Squeaks connected, objects in one can send messages to objects in the other, creating a distributed system. The Cog-based system now has access to the DOM of the web browser through ThisWebpage in the SqueakJS-based system, and the SqueakJS-based system has unsandboxed access to the host operating system through the Cog-based system. In particular, SqueakJS can use Cog to run network servers, so one can create a distributed system from SqueakJS instances running in many web browsers on many machines.
In tomorrow’s post, I’ll discuss the remote messaging protocol in more detail. After that, I’ll introduce a distributed network service that takes advantage of it.
a new platform
In particular, we can embed SqueakJS in a web page, and modify that web page from SqueakJS processes. It would be very useful to have a Smalltalk object model of the host web page. I have created such a thing with the new class ThisWebpage.
reaching out with ThisWebpage
I chose the name of ThisWebpage to be reminiscent of “thisContext”, the traditional Smalltalk pseudo-variable used by an expression to access its method execution context. In a similar way, expressions can use ThisWebpage to access the DOM of the hosting Web environment. One simple example is adding a button:
ThisWebpage createButtonLabeled: 'fullscreen' evaluating: [Project current fullscreen: true]
Behind the scenes, ThisWebpage is doing this:
(JS document createElement: 'input') at: #type put: 'button'; at: #onclick: put: [Project current fullscreen: true]
So far, ThisWebpage has some basic behavior for adding buttons and frames, and for referring to the document elements in which SqueakJS is embedded. It can also create links and synthesize clicks on them. This is an important ability, which I use in making a Squeak object memory jump from SqueakJS in a web browser to a native Cog virtual machine on the desktop (the subject of tomorrow’s post).
The possibilities here are immense. ThisWebpage is waiting for you to make it do amazing front-end things! Check it out as part of the Context 7 alpha 1 release.
I wrote a new website for Black Page Digital, my consultancy in Amsterdam and San Francisco. It features a running Squeak Smalltalk that you can use for livecoding. Please check it out, pass it on, and let me know what you think!