Archive for remote messaging

Livecoding other tabs with the Chrome Remote Debugging Protocol

Posted in Appsterdam, consulting, Context, Smalltalk, SqueakJS with tags , , , , , , , on 24 July 2017 by Craig Latta

Chrome Debugging Protocol

We’ve seen how to use Caffeine to livecode the webpage in which we’re running. With its support for the Chrome Remote Debugging Protocol (CRDP), we can also use it to livecode every other page loaded in the web browser.

Some Help From the Inside

To make this work, we need to coordinate with the Chrome runtime engine. For CRDP, there are two ways of doing this. One is to communicate using a WebSocket connection; I wrote about this last year. This is useful when the CRDP client and target pages are running in two different web browsers (possibly on two different machines), but with the downside of starting the target web browser in a special way (so that it starts a conventional webserver).

The other way, possible when both the CRDP client and target pages are in the same web browser, is to use a Chrome extension. The extension can communicate with the client page over an internal port object, created by the chrome.runtime API, and expose the CRDP APIs. The web browser need not be started in a special way, it just needs to have the extension installed. I’ve published a Caffeine Helper extension, available on the Chrome Web Store. Once installed, the extension coordinates communication between Caffeine and the CRDP.

Attaching to a Tab

In Caffeine, we create a connection to the extension by creating an instance of CaffeineExtension:

CaffeineExtension new inspect

As far as Chrome is concerned, Caffeine is now a debugger, just like the built-in DevTools. (In fact, the DevTools do what they do by using the very same CRDP APIs; they’re just another JavaScript application, like Caffeine is.) Let’s open a webpage in another tab, for us to manipulate. The Google homepage makes for a familiar example. We can attach to it, from the inspector we just opened, by evaluating:

self attachToTabWithTitle: 'Google'

Changing Feelings

Now let’s change something on the page. We’ll change the text of the “I’m Feeling Lucky” button. We can get a reference to it with:

tabs onlyOne find: 'Feeling'

When we attached to the tab, the tabs instance variable of our CaffeineExtension object got an instance of ChromeTab added to it. ChromeTabs provide a unified message interface to all the CRDP APIs, also known as domains. The DOM domain has a search function, which we can use to find the “I’m Feeling Lucky” button. The CaffeineExtension>>find: method which uses that function answers a collection of search results objects. Each search result object is a proxy for a JavaScript DOM object in the Google page, an instance of the ChromeRemoteObject class.

In the picture above, you can see an inspector on a ChromeRemoteObject corresponding to the “I’m Feeling Lucky” button, an HTMLInputElement DOM object. Like the JSObjectProxies we use to communicate with JavaScript objects in our own page, ChromeRemoteObjects support normal Smalltalk messaging, making the JavaScript DOM objects in our attached page seem like local Smalltalk objects. We only need to know which messages to send. In this case, we send the messages of HTMLInputElement.

As with the JavaScript objects of our own page, instead of having to look up external documentation for messages, we can use subclasses of JSObject to document them. In this case, we can use an instance of the JSObject subclass HTMLInputElement. Its proxy instance variable will be our ChromeRemoteObject instead of a JSObjectProxy.

For the first message to our remote HTMLInputElement, we’ll change the button label text, by changing the element’s value property:

self at: #value put: 'I''m Feeling Happy'

The Potential for Dynamic Web Development

The change we made happens immediately, just as if we had done it from the Chrome DevTools console. We’re taking advantage of JavaScript’s inherent livecoding nature, from an environment which can be much more comfortable and powerful than DevTools. The form of web applications need not be static files, although that’s a convenient intermediate form for webservers to deliver. With generalized messaging connectivity to the DOM of every page in a web browser, and with other web browsers, we have a far more powerful editing medium. Web applications are dynamic media when people are using them, and they can be that way when we develop them, too.

What shall we do next?



App streaming with Snowglobe

Posted in Appsterdam, consulting, Context, Naiad, Smalltalk, Spoon with tags , , , , , , , , on 31 October 2016 by Craig Latta

Now that we’ve seen how to run Smalltalk in a web browser, clone web Smalltalk as a desktop app, and send remote messages between Smalltalks, let’s look at an application of these technologies.

app streaming

App streaming is a way of delivering the user experience of an app without actually running the app on the user’s machine. The name is an allusion to music and video streaming; you get to experience the asset immediately, without waiting for it to download completely. Streaming an app also has the benefit of avoiding installation, something which can be problematic to do (and to undo). This is nice when you just want to demo an app, before deciding to install it.

Another advantage of app streaming is that the app can run on a much faster machine than the user’s, or even on a network of machines. Social networks are a crude example of app streaming; there are massive backends working behind your web browser, crunching away on all that graph data. Typically, though, app streaming involves an explicit visual component, with the user’s display and input devices standing in for the normal ones. The goal is to make using a new app as simple as playing an online video.

distributing Smalltalk user interface components

Everything in Smalltalk happens by objects sending messages to each other. With a remote messaging framework like Tether, we can put some of the objects in a user interface on a remote machine. Snowglobe is an adaptation of Squeak‘s Morphic user interface framework which runs Squeak on a server, but uses SqueakJS in a client web browser as the display. This is an easy way to recast a Smalltalk application as a web app, while retaining the processing speed and host platform access of the original.

Morphic is built around a display loop, where drawable components (morphs) are “stepped” at some frequency, like a flipbook animation. Normally, drawing is done on a single morph that corresponds to the display of the machine where Squeak is running. Snowglobe adds a second display morph which is Tether-aware. When drawing to this tethered display morph, the app server translates every display operation into a compact remote message.

To maximize speed, Morphic already tries to do its drawing with as few operations as possible (e.g., avoiding unnecessary redrawing). This is especially important when display operations become remote, since network transmission is orders of magnitude slower than local drawing. Since the tethered display morph also lives in a Smalltalk object memory, we can optimize drawing operations involving graphics that are known to both sides of the connection. For example, when changing the mouse cursor to a resize icon when hovering over the corner of a window, there’s no need to send the icon over the wire, since the displaying system already has it. Instead, we can send a much smaller message requesting that the icon be shown.

For full interaction, we also need to handle user input events going back the other way. Snowglobe co-opts Morphic’s user input handling as well. With user input and display forwarded appropriately together, we achieve the seamless illusion that our app is running locally, either as a single morph amongst other local morphs, or using the entire screen.

going beyond screen-sharing

Protocols like VNC do the remote display and user input handling we’ve discussed, although they are typically more complicated to start than clicking a link in a web browser. But since both systems in a Snowglobe session are Smalltalk, we can go beyond simple screen sharing. We can use Tether to send any remote messages we want, so either side can modify the app-streaming behavior in response to user actions. For example, the user might decide to go full-screen in the web browser displaying the app, prompting SqueakJS to notify the remote app, which could change the way the app displays itself.

try it for yourself

I’ve set up an AWS server running the Squeak IDE, reachable from SqueakJS in your web browser. Be gentle… there’s only one instance running (actually two, one in Europe and one in North America, chosen for you automatically by Amazon). Please check it out and let me know what you think!


Tether: remote messaging between Smalltalks with WebSockets

Posted in Appsterdam, consulting, Context, Naiad, Smalltalk, Spoon with tags , , , , , , on 30 October 2016 by Craig Latta

In my previous post, I introduced a new topology for distributed computation with Smalltalk: an object memory in SqueakJS in a web browser, paired by remote-messaging connection with another object memory in Cog in a native app, and connected with other SqueakJS/Cog pairs on other physical machines. The remote-messaging protocol that the memories speak is called Tether. I’ll go into a few details here.

passive frame-based messaging with WebSockets

We begin with a major constraint imposed by running in a web browser: we’re sandboxed, unable to listen for network connections. We must initiate a remote-messaging conversation by connecting to a listening server on a traditional operating system. Over the last few years, the W3C WebSockets standard has received widespread support in every major web browser. We can rely on the ability to create JavaScript WebSocket objects with the SqueakJS JavaScript bridge, and we can easily implement the WebSocket API in Smalltalk on non-web platforms using normal TCP sockets.

WebSockets use callback functions to deliver messages, or frames of bytes, between conversants. The Tether protocol imposes a structure on those bytes, which are processed on each side of the connection by instances of class Tether. The first four bytes are a 32-bit tag which indicates the Smalltalk class which should interpret the rest of the frame. In the case of a remote message, this is class Tether. Successive bytes indicate the message selector to perform, the receiver of the message, and the message parameters.

The message receiver is expressed as a 32-bit key into a table of exposed objects, maintained by the Tether instance handling the connection. The Tether instances themselves expose their identities to each other at the beginning of the conversation. Objects that aren’t specified by reference to an exposed-object table (such as message selectors) are expressed through serialization.

live serialization

The fact that both sides of the conversation are objects in live Smalltalk systems affords many optimizations that aren’t possible when serializing objects to a static file. For this reason, I call this live serialization. For example, when transferring a compiled method between systems, if the method’s literals are objects which already exist in the receiving system, we may write references to them rather than serializing them.

We can also take special measures when the receiving system is missing the classes whose instances we want to transfer. Instead of assuming in advance that the receiving system lacks the classes whose instances we’re transferring, and including them in our payload, as a static serialization file would, we can transfer such classes only on demand. This yields much higher accuracy in object transfer, and far fewer bytes sent over the wire. Since live serialization is part of a complete remote messaging protocol, any messages at all can be sent from either side to complete an object transfer.

With a receiver, selector, and parameters specified, the receiving system can perform the message sent from the sending system. Each object in the system is responsible for serializing itself over a Tether. If the answer to the remote message is a literal object, like a symbol or integer, it will write the bytes of its value on the Tether instance handling the message. If the answer isn’t a literal object, by default it will write a reference to itself. Developers can choose to pass objects by value or by reference as they see fit.


Tether performs every remote message in a distinct process, so that no system blocks waiting for an answer to be sent back over the network. Each remote message-send is assigned a unique identifier, and each answer is sent with the ID of its message-send as metadata, so that it can be delivered to the correct waiting process.

This scheme extends the traditional imperative messaging semantics of Smalltalk to any number of machines, and each message-send may involve receiver and parameter objects which are all on different machines. Every message-send may invoke any number of further remote messages before answering.

transparent proxies

An object which represents an object on a remote system is called a proxy. Ideally, it forwards every message sent to it to the remote object, and so provides the illusion of transparent remote messaging. Remote messaging in Smalltalk is often done by using a proxy class which inherits and implements as few messages as possible, and overriding the handler message sent by the virtual machine when a message is not understood. This provides enough coverage to do many useful things, but some messages handled specially by the virtual machine are not forwarded. Some use cases, like remote debugging, require forwarding even those special messages.

To achieve total forwarding coverage, we must modify the virtual machine. There are some situations where this is undesirable (e.g., a lack of tools or expertise, or a requirement to use a past virtual machine unmodified). Tether uses the “does not understand” tactic above in these situations, but provides a modified virtual machine for the rest. In this virtual machine, message forwarding is triggered during method lookup for instances of a specific proxy class (which can be located anywhere in the class hierarchy). Method caching and methods implemented directly as virtual machine instructions are also appropriately adapted. There are a few messages which proxies must understand locally, to participate in live serialization. These messages are also handled specially by the virtual machine.

see for yourself

Tether is an integral part of the Context project from Black Page Digital. A demo of remote messaging between SqueakJS and Cog is available. In tomorrow’s post, I’ll discuss an everyday application of remote messaging.


Caffeine: live web debugging with SqueakJS

Posted in Appsterdam, consulting, Context, Naiad, Smalltalk, Spoon with tags , , , , , , , , , , , , , , , , , , , , on 26 October 2016 by Craig Latta

In February 2015 I spoke about Bert Freudenberg’s SqueakJS at FOSDEM. We were all intrigued with the potential of this system to change both Smalltalk and web programming. This year I’ve had some time to pursue that potential, and the results so far are pretty exciting.

SqueakJS is a Squeak virtual machine implemented with pure JavaScript. It runs in all the web browsers, and features a bi-directional JavaScript bridge. You can invoke JavaScript functions from Smalltalk code, and pass Smalltalk blocks for JavaScript code to invoke as callbacks. This lets Smalltalk programmers take advantage of the myriad JavaScript frameworks available, as well as the extensive APIs exposed by the browsers themselves.

The most familiar built-in browser behavior is for manipulating the structure of rendered webpages (the Document Object Model, or “DOM”). Equally important is behavior for manipulating the operation of the browser itself. The Chrome Debugging Protocol is a set of JavaScript APIs for controlling every aspect of a web browser, over a WebSocket. The developer tools built into the Chrome browser are implemented using these APIs, and it’s likely that other browsers will follow.

Using the JavaScript bridge and the Chrome Debugging Protocol, I have SqueakJS controlling the web browser running it. SqueakJS can get a list of all the browser’s tabs, and control the execution of each tab, just like the built-in devtools can. Now we can use Squeak’s user interface for debugging and building webpages. We can have persistent inspectors on particular DOM elements, rather than having only the REPL console of the built-in tools. We can build DOM structures as Smalltalk object graphs, complete with scripted behavior.

I am also integrating my previous WebDAV work, so that webpages are manifested as virtual filesystems, and can be manipulated with traditional text editors and other file-oriented tools. I call this a metaphorical filesystem. It extends the livecoding ability of Smalltalk and JavaScript to the proverbial “favorite text editor”.

This all comes together in a project I call Caffeine. had fun demoing it at ESUG 2016 in Prague. Video to come…

new website for Black Page Digital

Posted in Appsterdam, consulting, Context, GLASS, music, Naiad, Seaside, Smalltalk, Spoon with tags , , , , , , , , , , , , , , , , on 21 January 2016 by Craig Latta

I wrote a new website for Black Page Digital, my consultancy in Amsterdam and San Francisco. It features a running Squeak Smalltalk that you can use for livecoding. Please check it out, pass it on, and let me know what you think!pano

new Context active filesystem layout

Posted in Appsterdam, consulting, Context, Naiad, Smalltalk, Spoon with tags , , , , , , , , , , , , , on 22 December 2014 by Craig Latta

When you start the Context app, you start a webserver that provides a “console”. Viewed through a host web browser, the console describes what Context is, and enables control of the memories it knows about. The webserver also provides an active filesystem via WebDAV. This lets you interact with the console from a host terminal or text editor, in a manner reminiscent of a Unix procfs (content is generated live-on-read). Here’s a typical filesystem layout, and what you can do with it:


            the idle process
               ProcessorScheduler class>>idleProcess
               [] in ProcessorScheduler class>>resume
            hello world

The README.html file is what the console displays initially. It has a directory sibling memories, containing a subdirectory for each memory the console knows about. Each memory is named by its UUID. In the session directory, there are files which give information about a memory. The state file looks like this:

# This memory is running. You can send it one of the following
# commands: snapshot, suspend, or stop. To do so, write this file with
# the desired command as the first word after this comment. Subsequent
# comments give other information about this memory, like host
# resource usage and virtual machine plugins loaded.

(type command here)

# host resource usage
# bytes used:        437,598
# bytes available: 1,328,467

# virtual machine plugins loaded
# FlowPlugin

In this way, a file in the active filesystem provides access to a read-eval-print loop (REPL). The user gives input to the console by writing the file; the console gives feedback to the user (including documentation) by generating appropriate content when the file is read.

The performance file looks like this:

# instructions per second: 382,184,269
# messages per second:      12,355,810

This gives general profiling information about the virtual machine.

The subdirectories of the classes directory correspond to the memory’s classes. Each one has subdirectories for its methods, subclasses, and metaclass. The methods directory has a file for each method of the class. This provides the ability to browse and change source code in the memory from a host text editor.

The processes directory has a subdirectory for each running process in the memory. Each process directory has a subdirectory for each context of that process. Each context directory has a REPL file for the source code of the context’s method, and a subdirectory for the context’s variables (including the context itself), each of which is an inspector in the form of a REPL file. In this way, much of the functionality of the traditional Smalltalk debugger is accessible from a host text editor.

Finally, the workspaces directory has subdirectories for multiple “workspaces”, where one may evaluate expressions and interact with their result objects. Each workspace has a source file, another REPL file which contains instructions, the expression to evaluate, and, on the next read after write, the textual form of the result. In addition, in a result directory, is a file named for the textual form of the result, containing a REPL inspector for that result object.

These tools are useful both for newcomers to live object systems who are more comfortable with a text editor than the Smalltalk GUI, and for those accessing systems running in the cloud, for which traditional GUI access might be awkward or prohibitive.

Smalltalk Reflections episode three is up

Posted in Appsterdam, consulting, Context, music, Smalltalk, Spoon with tags , , , , , , , , , , , , , , on 16 December 2014 by Craig Latta

Check it out!

%d bloggers like this: