Archive for livecoding

Pharo comes to Caffeine and SqueakJS

Posted in Appsterdam, consulting, Context, GLASS, Naiad, Seaside, Smalltalk, Spoon, SqueakJS with tags , , , , , , , , , on 29 June 2017 by Craig Latta

Google ChromeScreenSnapz025

The Caffeine web livecoding project has added Pharo to the list of Smalltalk distributions it runs with SqueakJS. Bert Freudenberg and I spent some time getting SqueakJS to run Pharo at ESUG 2016 in Prague last summer, and it mostly worked. I think Bert got a lot further since then, because now there are just a few Pharo primitives that need implementing. All I’ve had to do so far this time is a minor fix to the input event loop and add the JavaScript bridge. The bridge now works from Pharo, and it’s the first time I’ve seen that.

Next steps include getting the Tether remote messaging protocol and Snowglobe app streaming working between Pharo and Squeak, all running in SqueakJS. Of course, I’d like to see fluid code-sharing of all kinds between Squeak, Pharo, and all the other Smalltalk implementations.

So, let the bugfixing begin! :)  You can run it at https://caffeine.js.org/pharo/. Please do get in touch if you find and fix things. Thanks!

Backend Caffeine with Node.js and Tweetcoding

Posted in Appsterdam, consulting, Context, Smalltalk, Spoon, SqueakJS with tags , , , , , , on 27 June 2017 by Craig Latta

tweetcodingWe’ve seen a couple of examples of frontend Caffeine development, using jQuery UI and voxel.js. Let’s look at something on the backend, using Node.js and something I like to call tweetcoding.

Social Livecoding

Livecoding means having all the capabilities of your development environment while your system is live. Runtime is all the time. Smalltalk programmers have always enjoyed this way of working. It is particularly effective in domains such as music performance, where the ability to make changes while your system is running is critical, both for fixing bugs and for artistic expression. One of my artistic goals with musical livecoding (writing code that makes live music for an audience) is audience accessibility. I want the audience to have some understanding of what’s going on in my code, even if they aren’t coders (yet).

Another aspect I like to incorporate is audience participation. If the audience actually does understand what I’m doing in my code, why not let them join in? Social media has become a ubiquitous communication platform; billions of people are using it with prose and pictures. I think the time has come to use it with live code as well. Nearly everyone in the audience has a powerful computer with social media access in a pocket. We can all livecode together.

Let’s Do This

We need three things to make this happen: a way for people to submit code, something that listens for that code and evaluates it, and a shared artifact everyone can see that shows the code’s effects. I like Twitter as a coding medium, for its minimalism. It also has a culture of hashtag use, and a streaming API, which make listening for code straightforward. For the shared artifact, we can use something viewed with Caffeine in a web browser.

For our first example, let’s use something simple for our coding domain, like turtle graphics. A person can tweet instructions to a turtle which draws on a square piece of paper, 100 units on a side. There are four commands:

  • color, for the turtle to change its pen for one of the given color
  • down, for the turtle to put its pen down on the paper
  • rotate, for the turtle to change direction
  • move, for the turtle to move some distance
  • up, for the turtle to take its pen up from the paper

We’ll say that that the turtle can be identified by some ID, and is initially at the center of the paper. A tweetcoded instruction for the turtle might look like:

FranzScreenSnapz002

Tweetcoding tweets start with the #tweetcoding hashtag, so people seeing one for the first time can look at others with that hashtag, to see what it’s about. This is followed by hashtags identifying our turtle-graphics protocol, and the turtle we’re using. We end the tweet with a drawing instruction, using our instruction set above.

Livecoding NodeJS

To detect these tweets, we’ll use the Twitter streaming API from Node.js. In the spirit of livecoding, we won’t use an application-specific Node.js module for this directly, but instead inject JavaScript code live into a Node.js instance, from Caffeine in a web browser. I’ve written a node-livecode Node.js module which takes commands over a websocket. It starts with three instructions:

  • require, for loading another Node.js module into itself with npm
  • add instruction, for adding an instruction to itself
  • eval, for evaluating JavaScript code

You can see the implementation at GitHub. It can use SSL to keep injected code secure in transit, so you may want to set up a certificate for your server with Let’s Encrypt. Also note that it exposes the listening server as a global variable, so that you can use it in your injected code.

Once we have node-livecode on a server listening for commands, we can inject code into it from Caffeine. First we’ll inject code to listen to #tweetcoding tweets from Twitter:

| websocket |

websocket := JS WebSocket newWithParameters: {'wss://yourserver:port'}.
JS top at: #websocket put:websocket.

websocket
  at: #onmessage
  put: [:message |
    Transcript
      cr;
      nextPutAll: message data asString;
      endEntry];
  at: #onopen
  put: (
    (JS Function) new: '
      window.top.ws.send(JSON.stringify({
        credential: ''shared secret'',
        verb: ''require'',
        parameters: {
          package: ''node-tweet-stream'',
          then: ''
            var Twitter = require(''node-tweet-stream'')
              , twitter = new Twitter({
                  consumer_key: '''',
                  consumer_secret: '''',
                  token: '''',
                  token_secret: ''''})

            twitter.on(
              ''tweet'',
              function (tweet) {
                if (instructions[''broadcast'']) {
                  instructions[''broadcast''](tweet)}}

            twitter.on(
              ''error'',
              function (err) {
                console.log(''error in uploaded code'')})

            twitter.track(''#tweetcoding #turtlegraphics '')''}})))

The example above uses handwritten JavaScript code, but we could also use Caffeine’s JavaScript code generation to produce it from Smalltalk code (see the method MethodNode>>javaScript for the implementation). Also, since we’ll be injecting several things, it would be nice to have a more compact way of writing it. Let’s move that workspace code to a class.

Next we’ll inject a new instruction, for initializing a set of drawing-command listeners:

| client |

client := (
  NodeJSLivecodingClient
    at: 'wss://yourserver:port'
    withCredential: 'shared secret').

client
  addInstruction: 'initialize listeners'
  from: '
    function () {
      global.listeners = []
      return ''initialized listeners''}'

To round out our tweetcoding protocol, we’ll add instructions for accepting listeners, and broadcasting tweets detected from Twitter:

client
  addInstruction: 'start listening'
  from: '
    function () {
      listeners.push(this)
      return ''added listener''}'
  addInstruction: 'stop listening'
  from: '
    function () {
      listeners.splice(listeners.indexOf(this), 1);
      return ''removed listener''}'
  addInstruction: 'broadcast'
  from: '
    function (payload) {
      listeners.forEach(
        function (listener) {listener.send(payload)})
      return ''broadcasted''}'

Putting It All Together

Now we can write other Smalltalk classes which subscribe to tracked tweets from the server, and collaborate to do something interesting with them in the web browser. I’ve implemented the following classes:

Object
  NodeJSLivecodingClient
    TweetcodingClient
      TurtleGraphicsTweetcodingClient
  Turtle
  Tweet

Instances of NodeJSLivecodingClient can inject code into a node-livecoding server, and invoke code added by other clients. Instances of TweetcodingClient can also set tracking filters for tweets, and process matching tweets when they occur. Instances of TurtleGraphicsTweetcodingClient can also control Turtles, which can draw on canvases in Caffeine windows. Instances of Tweet bundle up the text and metadata of tweets. For the implementation, clear your browser’s cache for caffeine.js.org (including IndexedDB), and reload https://caffeine.js.org/.

I’m also running a node-livecoding server injected with turtle-graphics tweetcoding code, at wss://frankfurt.demo.blackpagedigital.com:8087/. Once you get connected (and tweets are flowing), you might see something like this:

Google ChromeScreenSnapz014

After developing this system, I’ve realized I don’t really need to run SqueakJS from within NodeJS; just injecting code into it is fine. There are many possibilities here. :)

Have fun, and please let me know what you think!

 

Caffeine :: Livecode the Web!

Posted in Appsterdam, consulting, Context, Naiad, Smalltalk, Spoon, SqueakJS with tags , , , , , , , , , , , , , , on 22 June 2017 by Craig Latta

CaffeineFor the impatient… here it is.

Back to the Future, Again

With the arrival of Bert Freudenberg’s SqueakJS, it was finally time for me to revisit the weird and wonderful world of JavaScript and web development. My previous experiences with it, in my consulting work, were marked by awkward development tools, chaotic frameworks, and scattered documentation. Since I ultimately rely on debuggers to make sense of things, my first question when evaluating a development environment is “What is debugging like?”

Since I’m a livecoder, I want my debugger to run in the web browser I’m using to view the site I’m debugging. The best in-browser debugger I’ve found, Chrome DevTools (CDT), is decent if you’re used to a command-line interface, but lacking as a GUI. With Smalltalk, I can open new windows to inspect objects, and keep them around as those objects evolve. CDT has an object explorer integrated into its read-eval-print loop (REPL), and a separate tab for inspecting DOM trees, but using them extensively means a lot of scrolling in the REPL (since asynchronous console messages show up there as well) and switching between tabs. CDT can fit compactly onto the screen with the subject website, but doesn’t make good use of real estate when it has more. This interrupts the flow of debugging and slows down development.

The Pieces Are All Here

With SqueakJS, and its JavaScript bridge, we can make something better. We can make an in-browser development environment that compares favorably with external environments like WebStorm. I started from a page like try.squeak.org. The first thing we need is a way to move the main SqueakJS HTML5 canvas around the page. I found jQuery UI to be good for this, with its “draggable” effect. While we’re at it, we can also put each of Squeak‘s Morphic windows onto a separate draggable canvas. This moves a lot of the computation burden from SqueakJS to the web browser, since SqueakJS no longer has to do window management. This is a big deal, since Morphic window management is the main thing making modern Squeak UIs feel slow in SqueakJS today.

SqueakJS provides a basic proxy class for JavaScript objects, called JSObjectProxy. Caffeine has an additional proxy class called JSObject, which provides additional reflection features, like enumerating the subject JS object’s properties. It’s also a good place for documenting the behavior of the JS objects you’re using. Rather than always hunting down the docs for HTMLCanvasElement.getContext on MDN, you can summarize things in a normal method comment, in your HTMLCanvasElement class in Smalltalk.

Multiple Worlds

With a basic window system based on HTML5 canvases, we can draw whatever we like on those canvases, using the SqueakJS bridge and whatever other JS frameworks we care to load. I’ve started integrating a few frameworks, including React (for single-page-app development), three.js (for WebGL 3D graphics development), and morphic.js (a standalone implementation of Morphic which is faster than what’s currently in Squeak). I’ll write about using them from Caffeine in future blog posts.

Another framework I’ve integrated into Caffeine is Snowglobe (for Smalltalk app streaming and other remote GUI access), which I wrote about here previously. I think the Snowglobe demo is a lot more compelling when run from Caffeine, since it can co-exist with other web apps in the same page. You can also run multiple Snowglobes easily, and drag things between them. I’ll write more about that, too.

Fitting Into the JavaScript Ecosystem

To get the full-featured debugger UI I wanted, I wrote a Chrome extension called Caffeine Helper, currently available on the Chrome Web Store. It exposes the Chrome Debugging Protocol (CDP) support in the web browser to SqueakJS, letting it do whatever the CDT can do (CDT, like SqueakJS, is just another JavaScript-powered web app). The support for CDP that I wrote about previously uses a WebSocket-based CDP API that requires Chrome to be started in a special way. The Caffeine Helper extension provides a JavaScript API, without that requirement.

I also wrote support for generating Smalltalk code from JavaScript, using the esprima parsing framework, and vice-versa. With my debugger and code generation, I’m going to try developing for some file-based JS projects, using Smalltalk behind the scenes and converting to and from JavaScript when necessary. I think JS web development might actually not drive me crazy this way. :)

Please Try It Out!

So, please check out Caffeine, at caffeine.js.org! I would very much appreciate your feedback. I’m particularly interested to hear your use cases, as I plan the next development steps. I would love to have collaborators, too. Let’s build!

Caffeine: live web debugging with SqueakJS

Posted in Appsterdam, consulting, Context, Naiad, Smalltalk, Spoon with tags , , , , , , , , , , , , , , , , , , , , on 26 October 2016 by Craig Latta

In February 2015 I spoke about Bert Freudenberg’s SqueakJS at FOSDEM. We were all intrigued with the potential of this system to change both Smalltalk and web programming. This year I’ve had some time to pursue that potential, and the results so far are pretty exciting.

SqueakJS is a Squeak virtual machine implemented with pure JavaScript. It runs in all the web browsers, and features a bi-directional JavaScript bridge. You can invoke JavaScript functions from Smalltalk code, and pass Smalltalk blocks for JavaScript code to invoke as callbacks. This lets Smalltalk programmers take advantage of the myriad JavaScript frameworks available, as well as the extensive APIs exposed by the browsers themselves.

The most familiar built-in browser behavior is for manipulating the structure of rendered webpages (the Document Object Model, or “DOM”). Equally important is behavior for manipulating the operation of the browser itself. The Chrome Debugging Protocol is a set of JavaScript APIs for controlling every aspect of a web browser, over a WebSocket. The developer tools built into the Chrome browser are implemented using these APIs, and it’s likely that other browsers will follow.

Using the JavaScript bridge and the Chrome Debugging Protocol, I have SqueakJS controlling the web browser running it. SqueakJS can get a list of all the browser’s tabs, and control the execution of each tab, just like the built-in devtools can. Now we can use Squeak’s user interface for debugging and building webpages. We can have persistent inspectors on particular DOM elements, rather than having only the REPL console of the built-in tools. We can build DOM structures as Smalltalk object graphs, complete with scripted behavior.

I am also integrating my previous WebDAV work, so that webpages are manifested as virtual filesystems, and can be manipulated with traditional text editors and other file-oriented tools. I call this a metaphorical filesystem. It extends the livecoding ability of Smalltalk and JavaScript to the proverbial “favorite text editor”.

This all comes together in a project I call Caffeine. had fun demoing it at ESUG 2016 in Prague. Video to come…

new website for Black Page Digital

Posted in Appsterdam, consulting, Context, GLASS, music, Naiad, Seaside, Smalltalk, Spoon with tags , , , , , , , , , , , , , , , , on 21 January 2016 by Craig Latta

I wrote a new website for Black Page Digital, my consultancy in Amsterdam and San Francisco. It features a running Squeak Smalltalk that you can use for livecoding. Please check it out, pass it on, and let me know what you think!pano

new Context active filesystem layout

Posted in Appsterdam, consulting, Context, Naiad, Smalltalk, Spoon with tags , , , , , , , , , , , , , on 22 December 2014 by Craig Latta

When you start the Context app, you start a webserver that provides a “console”. Viewed through a host web browser, the console describes what Context is, and enables control of the memories it knows about. The webserver also provides an active filesystem via WebDAV. This lets you interact with the console from a host terminal or text editor, in a manner reminiscent of a Unix procfs (content is generated live-on-read). Here’s a typical filesystem layout, and what you can do with it:

/
   README.html

   memories
      3EAD9A45-F65F-445F-89C1-4CA0A9D5C2F8
         session
            state
            performance
         classes
            Object
               metaclass
                  (etc.)
               methods
                  at:
                  (etc.)
               slots
                  all
                     (etc.)
                  inherited
                     (etc.)
                  local
                     (etc.)
               subclasses
                  (etc.)
         processes
            the idle process
               ProcessorScheduler class>>idleProcess
                  source
                  variables
                     thisContext
                     self
                     (etc.)
               [] in ProcessorScheduler class>>resume
               (etc.)
            (etc.)
         workspaces
            hello world
               source
               result
                  7

The README.html file is what the console displays initially. It has a directory sibling memories, containing a subdirectory for each memory the console knows about. Each memory is named by its UUID. In the session directory, there are files which give information about a memory. The state file looks like this:

# This memory is running. You can send it one of the following
# commands: snapshot, suspend, or stop. To do so, write this file with
# the desired command as the first word after this comment. Subsequent
# comments give other information about this memory, like host
# resource usage and virtual machine plugins loaded.

(type command here)

# host resource usage
#
# bytes used:        437,598
# bytes available: 1,328,467

# virtual machine plugins loaded
#
# FlowPlugin

In this way, a file in the active filesystem provides access to a read-eval-print loop (REPL). The user gives input to the console by writing the file; the console gives feedback to the user (including documentation) by generating appropriate content when the file is read.

The performance file looks like this:

# instructions per second: 382,184,269
# messages per second:      12,355,810

This gives general profiling information about the virtual machine.

The subdirectories of the classes directory correspond to the memory’s classes. Each one has subdirectories for its methods, subclasses, and metaclass. The methods directory has a file for each method of the class. This provides the ability to browse and change source code in the memory from a host text editor.

The processes directory has a subdirectory for each running process in the memory. Each process directory has a subdirectory for each context of that process. Each context directory has a REPL file for the source code of the context’s method, and a subdirectory for the context’s variables (including the context itself), each of which is an inspector in the form of a REPL file. In this way, much of the functionality of the traditional Smalltalk debugger is accessible from a host text editor.

Finally, the workspaces directory has subdirectories for multiple “workspaces”, where one may evaluate expressions and interact with their result objects. Each workspace has a source file, another REPL file which contains instructions, the expression to evaluate, and, on the next read after write, the textual form of the result. In addition, in a result directory, is a file named for the textual form of the result, containing a REPL inspector for that result object.

These tools are useful both for newcomers to live object systems who are more comfortable with a text editor than the Smalltalk GUI, and for those accessing systems running in the cloud, for which traditional GUI access might be awkward or prohibitive.

Smalltalk Reflections episode three is up

Posted in Appsterdam, consulting, Context, music, Smalltalk, Spoon with tags , , , , , , , , , , , , , , on 16 December 2014 by Craig Latta

Check it out!

%d bloggers like this: