Archive for minimalism

Caffeine: live web debugging with SqueakJS

Posted in Appsterdam, consulting, Context, Naiad, Smalltalk, Spoon with tags , , , , , , , , , , , , , , , , , , , , on 26 October 2016 by Craig Latta

In February 2015 I spoke about Bert Freudenberg’s SqueakJS at FOSDEM. We were all intrigued with the potential of this system to change both Smalltalk and web programming. This year I’ve had some time to pursue that potential, and the results so far are pretty exciting.

SqueakJS is a Squeak virtual machine implemented with pure JavaScript. It runs in all the web browsers, and features a bi-directional JavaScript bridge. You can invoke JavaScript functions from Smalltalk code, and pass Smalltalk blocks for JavaScript code to invoke as callbacks. This lets Smalltalk programmers take advantage of the myriad JavaScript frameworks available, as well as the extensive APIs exposed by the browsers themselves.

The most familiar built-in browser behavior is for manipulating the structure of rendered webpages (the Document Object Model, or “DOM”). Equally important is behavior for manipulating the operation of the browser itself. The Chrome Debugging Protocol is a set of JavaScript APIs for controlling every aspect of a web browser, over a WebSocket. The developer tools built into the Chrome browser are implemented using these APIs, and it’s likely that other browsers will follow.

Using the JavaScript bridge and the Chrome Debugging Protocol, I have SqueakJS controlling the web browser running it. SqueakJS can get a list of all the browser’s tabs, and control the execution of each tab, just like the built-in devtools can. Now we can use Squeak’s user interface for debugging and building webpages. We can have persistent inspectors on particular DOM elements, rather than having only the REPL console of the built-in tools. We can build DOM structures as Smalltalk object graphs, complete with scripted behavior.

I am also integrating my previous WebDAV work, so that webpages are manifested as virtual filesystems, and can be manipulated with traditional text editors and other file-oriented tools. I call this a metaphorical filesystem. It extends the livecoding ability of Smalltalk and JavaScript to the proverbial “favorite text editor”.

This all comes together in a project I call Caffeine. had fun demoing it at ESUG 2016 in Prague. Video to come…

new website for Black Page Digital

Posted in Appsterdam, consulting, Context, GLASS, music, Naiad, Seaside, Smalltalk, Spoon with tags , , , , , , , , , , , , , , , , on 21 January 2016 by Craig Latta

I wrote a new website for Black Page Digital, my consultancy in Amsterdam and San Francisco. It features a running Squeak Smalltalk that you can use for livecoding. Please check it out, pass it on, and let me know what you think!pano

new Context active filesystem layout

Posted in Appsterdam, consulting, Context, Naiad, Smalltalk, Spoon with tags , , , , , , , , , , , , , on 22 December 2014 by Craig Latta

When you start the Context app, you start a webserver that provides a “console”. Viewed through a host web browser, the console describes what Context is, and enables control of the memories it knows about. The webserver also provides an active filesystem via WebDAV. This lets you interact with the console from a host terminal or text editor, in a manner reminiscent of a Unix procfs (content is generated live-on-read). Here’s a typical filesystem layout, and what you can do with it:

/
   README.html

   memories
      3EAD9A45-F65F-445F-89C1-4CA0A9D5C2F8
         session
            state
            performance
         classes
            Object
               metaclass
                  (etc.)
               methods
                  at:
                  (etc.)
               slots
                  all
                     (etc.)
                  inherited
                     (etc.)
                  local
                     (etc.)
               subclasses
                  (etc.)
         processes
            the idle process
               ProcessorScheduler class>>idleProcess
                  source
                  variables
                     thisContext
                     self
                     (etc.)
               [] in ProcessorScheduler class>>resume
               (etc.)
            (etc.)
         workspaces
            hello world
               source
               result
                  7

The README.html file is what the console displays initially. It has a directory sibling memories, containing a subdirectory for each memory the console knows about. Each memory is named by its UUID. In the session directory, there are files which give information about a memory. The state file looks like this:

# This memory is running. You can send it one of the following
# commands: snapshot, suspend, or stop. To do so, write this file with
# the desired command as the first word after this comment. Subsequent
# comments give other information about this memory, like host
# resource usage and virtual machine plugins loaded.

(type command here)

# host resource usage
#
# bytes used:        437,598
# bytes available: 1,328,467

# virtual machine plugins loaded
#
# FlowPlugin

In this way, a file in the active filesystem provides access to a read-eval-print loop (REPL). The user gives input to the console by writing the file; the console gives feedback to the user (including documentation) by generating appropriate content when the file is read.

The performance file looks like this:

# instructions per second: 382,184,269
# messages per second:      12,355,810

This gives general profiling information about the virtual machine.

The subdirectories of the classes directory correspond to the memory’s classes. Each one has subdirectories for its methods, subclasses, and metaclass. The methods directory has a file for each method of the class. This provides the ability to browse and change source code in the memory from a host text editor.

The processes directory has a subdirectory for each running process in the memory. Each process directory has a subdirectory for each context of that process. Each context directory has a REPL file for the source code of the context’s method, and a subdirectory for the context’s variables (including the context itself), each of which is an inspector in the form of a REPL file. In this way, much of the functionality of the traditional Smalltalk debugger is accessible from a host text editor.

Finally, the workspaces directory has subdirectories for multiple “workspaces”, where one may evaluate expressions and interact with their result objects. Each workspace has a source file, another REPL file which contains instructions, the expression to evaluate, and, on the next read after write, the textual form of the result. In addition, in a result directory, is a file named for the textual form of the result, containing a REPL inspector for that result object.

These tools are useful both for newcomers to live object systems who are more comfortable with a text editor than the Smalltalk GUI, and for those accessing systems running in the cloud, for which traditional GUI access might be awkward or prohibitive.

Smalltalk Reflections episode three is up

Posted in Appsterdam, consulting, Context, music, Smalltalk, Spoon with tags , , , , , , , , , , , , , , on 16 December 2014 by Craig Latta

Check it out!

Context release 4 alpha 1

Posted in Appsterdam, consulting, Context, Naiad, Smalltalk, Spoon with tags , , , , , , on 9 December 2014 by Craig Latta

Context 4 alpha 1 is released. This one fixes loading errors in the welcome page, supports remote debugging and process browsing, and makes Naiad and remote messaging support available as Monticello packages. Thanks in advance for reporting bugs!

a detailed description of Context’s object memory visualization tools

Posted in Appsterdam, consulting, Context, Smalltalk, Spoon with tags , , , , , , , on 22 October 2012 by Craig Latta

direct graphical mapping of the object memory

When I got the current Context snapshot down to about 450 kilobytes, I realized that I could render it as a reasonably-sized picture (within 800 by 600 pixels). I was interested in doing this mostly because I’d never seen a graphical representation of the Smalltalk object memory before. As I proceeded, I found it enabled some pretty useful things.

simulator movies

In Squeak, I made an 8-bit-per-pixel bitmap, put the snapshot bytes into it linearly, and displayed it. It looked like a very peculiar sort of confetti… It wasn’t hard to see a few vague patterns, though. I instrumented the Squeak virtual machine simulator so that it would stop every so often to render a picture. I also chose the color of each pixel based on the class of the object of the corresponding byte.

I ran the simulator through the first 3000 instructions a Context object memory goes through after resuming, stopping after every 50 instructions to render a picture. (The simulated Context system went through its startup routine of reconnecting over the network to a headful system, so that one may browse it via remote messaging.) I put the pictures together into a 30-frames-per-second movie, a new picture every four frames. So here’s an example of extreme slow-motion; in the course of two seconds one sees roughly the first eight microseconds of execution time (as measured on my hardware).

exploring the space

But why just look at the pictures? I changed the normal picture viewer event handler so that when you click on a pixel, a cached simulator tells you more about the object of which the corresponding byte is a part. Coupled with an on-screen magnifying glass, this has turned out to be a valuable tool for finding more objects to delete from the Context snapshot.

Finally, I augmented the simulator to export graphs of all the reference relationships between all the objects in the system. Rummaging around in them in 3D is a lot of fun.

Please feel free to explore the graph data yourself and let me know if you find anything interesting. :) There are 8,628 objects (labelled) and 28,449 connections. It’s in a format for use with the Walrus graph visualizer (a lovely thing!).

I think this stuff gives new meaning to the old jargon phrase “smalltalk image”. :)

a detailed Naiad description

Posted in Appsterdam, consulting, Naiad, Smalltalk, Spoon with tags , , , , , , , , , on 21 October 2012 by Craig Latta

(from the Spoon release notes)

Spoon is a project to make Smalltalk systems more understandable, by removing unnecessary stuff, reorganizing what remains, and making it easier to share and deploy. If you’re interested in teaching, modules, bootstrapping, minimalism, remote messaging, proxies, metaprogramming, streams, sockets, or namespaces, you may be interested in this.

Motivation

Smalltalk began as a part of a vision to provide extraordinary computing power to creative individuals. Although computer networks grew alongside it, Smalltalk was released into the world while its support for collaboration was still rudimentary. The system uses the concept of a virtual machine, including a model of the machine’s memory, composed entirely of objects. Besides providing a very portable execution model, this scheme gives the system a powerful continuity: everything happens as the result of sending messages to objects. When it comes to sharing objects with other people, however, this continuity is violated.

Traditionally, to share code with another Smalltalk programmer, one writes source code to a file, and shares the file. One is no longer sending messages to objects, and the behavior being shared is reduced from live objects to a textual form which requires compilation to reanimate. When it is in that form, one cannot ask it to do things, as with objects. Worse, source code is ambiguous; the result of compilation depends on the environment in which it takes place. It’s easy for the target compilation environment to be different from the original one in very subtle ways. Even referring to a class by name is risky; there is no way to be sure that the class with that name in the target compilation environment (if there even is one) is equivalent to the one in the original.

With the means of code-sharing being relatively unexpressive and inconvenient, Smalltalk object memories develop mostly in isolation, and quickly diverge from one another. And within each memory, support for describing the component behavior of different subsystems is weak. The result, after over thirty years of evolution, is that it is difficult even to describe what behavior one wants to transfer, before subjecting it to the inaccuracies of the transfer itself. Object memories have grown into large knots which are hard to untangle manually. Smalltalk’s designers pursued a system that one person could understand and maintain. These problems hinder that pursuit, and make working in teams much harder than it could be. They make the experience of teaching and learning the system awkward, and this is one reason why the Smalltalk community is small.

Approach

Smalltalk programmers who persevere in spite of these problems live with them for quite a while. They often don’t even notice them, both because the system in general is so much more powerful and fun than those they have used before, and because the peculiarities of the development process are longstanding traditions. When it becomes too painful to ignore, many people contemplate something better, but end up using workarounds that seem to make a reasonable trade between improvement and compatibility. There always seem to be more important things to work on. I think the workarounds created so far have made team development tolerable (for example, Monticello and SqueakSource), but they don’t make the system much easier to understand.

For me, the point at which the pain became intolerable came while I was teaching a Squeak course at Stanford University. There’s nothing like explaining the system to a room full of inquisitive college students to make its shortcomings clear. You see the system with new eyes, and a fresh sensibility. In order to explain one concept, I had to explain several interdependent pieces of folklore. I wanted to show how object-oriented programming enables powerful divisions of responsibility, but it was tricky to do this with Smalltalk as my exemplar.

I decided it was time for the extraneous stuff to go. I wanted to know what is truly essential for a functioning Smalltalk system, so I could show it to others. Once I had identified that, I wanted a way to compose larger systems while avoiding the knots of the past.

Nuke It From Space; It’s The Only Way To Be Sure

I started by creating a proxy system and a remote system browser that used it, so that I could remove things from afar. As anyone who has tried to refactor the user interface knows, it’s tricky to do that while using it. By changing the system remotely, I could use another system’s user interface instead. I was able to remove large swathes of the system with impunity, which was fun.

I got pretty far with this before I realized there’s a better way to discard unused objects: throw away all the methods that haven’t been run recently. I modified the virtual machine to set a mark bit in each method it runs, and wrote a version of the garbage collector which treats unmarked methods as garbage. I could clear the marks on all the methods in the system, then run the system for a while (ideally, run unit tests). When I ran the new collector, all the unmarked methods got discarded.

More importantly, all the literals of those methods got discarded, if there was no other path from them to some marked method. By running this from a remote browser while the target system ran headlessly, I was able to discard the entire graphics system, including Morphic, in one stroke. This led to a series of small object memories, the smallest being 1,337 bytes long (it adds 3 and 4 and then exits).

Visualization

I realized I now had memories with fewer bytes than there were pixels on my screen. I wanted to see what they looked like. I extended Squeak’s virtual machine simulator to write the object memory as a picture, one pixel per object byte, with the color of each pixel determined by the class of the corresponding object. By writing such a picture every few instructions, I made animations that showed contexts being created, and objects allocated and reclaimed. I made an interactive browser that could magnify a region of one picture, and print detailed information about the object corresponding to a pixel chosen with the mouse.

I also wanted a visualization of the references between objects. Again I extended the simulator, to output a directed graph of all object references at a single moment in time, readable by the Walrus interactive hyperbolic graph viewer. This provided the most compelling illustration yet of a Smalltalk memory’s contents. Check out the demonstrations of all these visualization tools.

Imprinting

At the same time I was ripping things out, I was using remote messaging to put things in. I designed a way to transfer compiled methods from one system to another directly. I call this imprinting. I extended my remote browser to compile source code in the user-interface memory, and transfer the results directly to the remote memory. With this direct transfer, the compilation environment of the target memory matters much less, because no compilation is necessary in the target memory. If you can transfer a method’s literals and instructions correctly, then source code becomes an optional piece of documentation for the benefit of humans. In Spoon, literal markers describe method literals. They are transferred instead of the literals themselves, and they can recreate those literals when their methods are installed.

Transferring some method literals, like strings, is trivial. Others pose a challenge. In particular, how should one transfer a class reference? If a class object in one system and the corresponding class object in another system are truly equivalent in how they define the state of their instances, then we can refer to them using a single unique identifier. Like source code generally, class names become an optional piece of documentation. Compiled methods, and the virtual machines that run them, don’t care what the names of the classes are. It is only important that the instances the methods manipulate are of the expected storage format. This led to the namesake concept of Spoon’s module system.

NAIAD: Name and Identity Are Distinct

When you compile a method from source code that refers to a class, that method object has a reference to a unique live class object in the system. By using a unique identifier to refer to that class, rather than a textual name, we can preserve the identity of that class object across multiple object memories. By transferring methods directly, without resorting to recompilation, we can preserve the behavior of the instances of that class across memories too.

Naiad, Spoon’s module system, provides a framework for structured imprinting. With it, we can not only transfer methods, but also refer to particular versions of execution environment components, including methods, modules (groups of methods), classes, authors, comments, tags, and checkpoints. We can create collections of these references, called edit histories, that can answer important questions about the execution environment over time.

Identifiers and Editions

The basis for these references is the identifier, or ID. Each class in the system (both meta and non-meta) is given a universally unique identifer, or UUID, also known as its base ID. Add to that the UUID of the author who defined the class, and a version, and you have a class ID. Add to that the selector, version, and author UUID of a method of that class, and you have a method ID.

With these IDs, we can record the act of installing the corresponding components, as editions. An edition is a description of a component, sufficient to create an identical instance of that component in another memory. For example, a method edition records the header, literal markers, initial and ending program counter values, instructions, and class edition of a method, and a class edition records the format, superclass ID, and method editions of a class.

A class in Spoon still has a name, but it is part of the state of that class, along with its base ID. Each class is responsible for its own name, and there is no longer a need for a central registry of names. In Spoon there is no system dictionary. The compiler asks a class directly for its name binding. The system roots, including class Object, are known to the compiler. One could argue that, apart from the pseudovariables (true, false, nil, and thisContext), there are no global variables in Spoon.

Edit Histories and Modules

The place where we record editions is an edit history. Here editions are associated with the IDs of their components. We can then look up the edition for any component, given its ID. For example, we can ask a class for its base ID, and ask the edit history for the active class edition for that base ID. Editions have references to the previous and next editions timewise for their component, so we can make more complex queries such as “What are classes in the system that have ever had a method version written by this author?”

Modules are collections of method IDs. A module transfers the methods it describes to a remote memory by creating a copy of itself in the remote system, and guiding that system through a synchronization of the two memories. By “synchronization”, I mean that the original module is smart about not installing components which are already present in the remote memory. Many of the questions that an original module will ask its remote copy are ultimately answered by the edit history of the module copy’s system.

The edit history also augments the traditional Smalltalk source files (“changes” and “sources”). For a given development system, it occupies its own object memory, called a history memory, maintaining a remote messaging connection to the memory it describes, called the subject memory. It’s feasible to give the edit history its own memory because memories are now small by default.

Note that in cases where live system synchronization is not possible due to connectivity constraints, one can still send an entire history memory asynchronously to the requestor and let the synchronization happen locally. Similarly, the history memory that a developer uses can be located on any connected machine. For example, one could use a subject memory on an iPhone that has its history memory in the cloud.

Module Discovery

Finally, Naiad provides a means of finding module objects that one would like to install. It uses Google to index web pages that describe all available modules. Each module page includes a link to a local Spoon web server that, when followed, instructs the local Spoon system about how to install the module.

You can read about the Naiad object model in greater detail elsewhere on my blog. And again, all the Naiad classes are here in the system you can run now.

Future Work

There are at least two significant things missing from the current release. The first is a reorganization of the virtual machine, which is itself another sprawling and confusing knot (although the least so of the Smalltalk VMs I’ve encountered). The second is support for secure messaging (unless one uses an end-to-end scheme based on SSH tunnels or a VPN). The remote messaging system here is a minimal bootstrapping measure. Clearly one can imagine using something more sophisticated, like TeaTime. Again, though, I think a major simplification would be in order.

I want to extend Spoon beyond Squeak to other Smalltalk implementations. That’s my project for Camp Smalltalk at various conferences. I’m available to apply this work in paid and academic contexts. I’m currently a consultant based in Amsterdam, eager to work with clients worldwide. I would also like to pursue a PhD degree based on this work. I would love to hear from you.

Thanks

Thanks for reading this far, or just being interested enough to see what’s at the end. :) If you have any questions, please feel free to ask! I am usually on the Spoon IRC channel between 9am and 9pm GMT.

%d bloggers like this: