<p>An incomplete collection of interactive projects that I&apos;ve been involved in or engineered personally. Most of these are from before I was at PlayStation, during my decade at the agency <a href="https://stimulant.com">Stimulant</a>. Scattered among those are small free-time side-quests.</p>

I've been a fan of the work of <a href="https://www.thedesignersrepublic.com">The Designers Republic</a> since forever, and became intrigued when they opened up a Discord server and <a href="https://www.tdrbingo.com">NFT project</a>. I was <a href="https://opensea.io/assets/ethereum/0xcb6b570b8aeabe38b449aff31f901b8e1b91e396/4">the first to mint</a>, but the project had a slow and mysterious reveal. Members of the community started to piece together how the tokens were related and came up with a manually-created pyramid visualization. Of course I had to build it in code, combining the high-resolution artwork into a ~9 gigapixel image pyramid. The best parts are the keyboard navigation between tokens, and the responsive mobile layout.

NFT visualization

TDRBingo® Visualization

The kitchen tile in my home is a pattern called "penny round" that seemed made for projecting on. I developed a calibration system and drawing tool using React and <a href="https://two.js.org">two.js</a>, then animated a "show" using <a href="https://greensock.com/gsap/">GSAP</a>. The algorithm computes all possible lines through the area from the tiles that intersect with the outer cabinets. A very basic <a href="https://www.viewsonic.com/us/m1mini-ultra-portable-led-projector-with-jbl-speaker-hdmi-and-usb.html#specs">ViewSonic M1 projector</a> was used.

projection mapping at home with React

Pennyround

An interactive exploration of Sol LeWitt's "Drawing Series", one of the groups of his many Wall Drawings, produced beginning in 1960. I wrote a script which procedurally generated all of the drawings as SVG, rasterized them to massive bitmaps, and split them into image pyramids to be navigated with <a href="https://openlayers.org">OpenLayers</a>. The drawing methods, styles, and series can be browsed spatially. Each drawing can also be viewed on its own, with links back to the overall map.

exploration of a generative system

Sol LeWitt: Drawing Series

After my first trip to Miami for Art Basel and the other fairs, for some reason I was compelled to visualization of the trip and detailed description of all the stops we made. This was based on a scrape of my <a href="https://www.swarmapp.com">Swarm</a> check-ins across the week.

overengineered travel blog

A First-Timer's Guide to Miami Art Week

A platform on which to build experiences which respond to guest interaction in physical space. Multiple sensing technologies including depth/RGB cameras, beacons, and UWB are aggregated into a single coordinate space and event stream. A guests' session can begin with a website click, persist when they enter a physical space, and influence immersive installations. Implemented as decoupled micro-services which can theoretically scale to an infinitely large physical space.

technology-agnostic tracking of guest interactions

Experiential Sensing Platform

A high-resolution Bluetooth sensor network was installed throughout a full-scale, high-resolution mockup of a retail showroom for a major automotive brand. A series of seven touch screen exhibits guided visitors through a purchase, with BLE tracking enabling a persistent session across each one. A mobile exhibit reflected information about a vehicle as guests walked around it. Administrators could view reports and heat maps of guest journeys over time.

physical guest tracking across digital exhibits

Prototype Retail Space

I love digital and installation art, but wanted to enjoy it at home. I developed a digital signage system which accepts links to videos from around the web and plays them offline. Content can be grouped into channels with multiple displays playing the same or different channels. A physical rotary controller allows for seeking, skipping, muting, and displaying metadata.

in-home digital art display with a physical controller

RGBTV

A series of exhibits on the outside and inside of a custom fiberglass form. Three touch screens present a network of related stories, each on a given theme. Stories display on top of an ambient background which is synchronized between the touch screens and the LEDs spanning the spaces between them. The meditative space within the sculpture presents a <a href="https://en.wikipedia.org/wiki/Wish_tree">wishing tree</a> rendered in realtime 3D, which is also available on the museum website. All content is editable by museum staff by via a third-party CMS installation.

touch screens, LED effects, 3D microsite, CMS

City of Hope Museum

Approved photos with the event's hashtag were scraped from social media and merged into a Delaunay collage before being twisted and warped into a mesh of triangles.

distorted social media viewer for a festival

Meta Mirror

The brand's innovation team had a seed of an idea which we expanded to a physical prototype and a suite of applications to run on it. High-quality video streams from DSLR cameras were merged with depth data to create large-scale, headset-free augmented reality experiences. Users interacted with voice, touch, gesture, and mobile devices.

headset-free augmented reality

Prototype Consumer Device

We designed and implemented a 360° projection theater which tracked visitors via a series of cameras in the ceiling and responded with 3D animations and LED lighting. An adjacent disused elevator shaft reflected visitor silhouettes as a swirl of particles for selfies. In between, an infinity mirror lit up when guests walked on it, with custom lighting schemes for every occasion. A custom show control system tied it all together and let staff run the exhibits from a tablet.

360° projection, motion tracking, responsive lighting

OUE Skyspace LA

Ringmaster is a show control system in which hardware and software to be controlled are defined in a form of markup and organized into a hierarchy. Any number of devices such as projectors, displays, computers, and lighting controllers can be defined. A mobile-friendly dashboard displays device status and issues commands. Commands can also be issued on a schedule, and alerts can be sent automatically if things go wrong. Operated continuously for years at <a href="/project/skyspace">OUE Skyspace</a>.

automate control of hardware and software

Ringmaster

We designed and implemented large series of exhibits to educate visitors on the history of Intel and the chip manufacturing process. The software platform evolved over the years, from Silverlight to Unity to Cinder. Some exhibits were augmented with presence-sensing depth cameras, physical controls and buttons, and content moderation tools.

five years, ten exhibits

Intel Corporate Museum

A series of touch screen kiosks which let visitors pan and zoom around a massive 360° image of Philadelphia. Day and night images are synced together and dotted with points of interest organized by category, each with copy and a slide show. All text and UI is available in multiple languages.

zoom into the city

Philadelphia Gigapixel Zoom

This touch screen digital signage system on Genentech’s campus is running in seven building lobbies, each with a different configuration of displays and touch sensors. The content is a blend of items from Twitter, building sensors, and the company intranet.

news and information in many form factors

Genentech Lobby Displays

A 65' by 16' projection greets visitors to the tower by highlighting their origin on a 3D world map and welcoming them in their native language. The experience consists of a Cinder application running at 8k resolution and a Node.js server which gets ticket sale and scan data in real time. Millions of visitors have been tracked by the system so far.

ticket sales mapped on a large projection

Visitor Map at One World Trade

A series of touch screen kiosks which let visitors pan and zoom around a massive 360° image of Seattle, dotted with points of interest organized by category, each with copy and a slide show. A second experience teleports guests to insider locations around the city with immersive video.

teleport around the city from above

Space Needle City Explorers

<a href="https://github.com/stimulant/ampm">An open-source project</a> to perform application management and performance monitoring for Stimulant’s installation projects. The tool launches application processes, restarts them if there’s an error, and collects logging and analytics information. Deployed in many Stimulant projects and used by other agencies as well.

application monitoring and performance management

am/pm

Microsoft created a sculpture that was projected on from within, including integrated Kinect sensors on four sides. We visualized the skeletons of dancers on all sides, connecting the limbs of those on opposite sides of the cube. It was installed in the main venue of a Seattle music festival, and reacted to the audio in real-time.

audio-reactive dance installation

The Microsoft Cube

This wide-screen display is behind a mirrored surface and augmented with depth cameras, reflecting the guest's image and a particle silhouette as they approach to play with a number of mini-games that inspire innovation and collaboration.

digital mirror to inspire innovation

Launchpad Exhibit

A transparent LCD display, a laptop behind it, and a mapped projection all display synchronized product information when visitors interact. An integrated calibration system is included to adjust the projection as needed.

transparent display and projection mapping

Intel Ultrabook Exhibit

Deployed across multiple venues during a Seattle music festival, the signage displayed upcoming performance schedules and social media content scraped from the web. Organizers could inject announcements as the event went on. A series of generative backgrounds reacted in realtime to the music.

reactive visuals, social media, and show times

Decibel Festival Digital Schedule

A six-screen touch wall in the lobby of a chip manufacturer’s new corporate headquarters included generative and touch-reactive graphics as an ambient mode, and a playful way to navigate content describing the company’s history and products.

generative effects and company info

Maxim Lobby Touch Wall

Conference attendees could use any of six custom-made stations to scan an object. From this silhouette, a unique life form was created on a 2200 square-foot, non-uniform, projection-mapped surface, using 24 projectors.

object scanning and projection mapping

Intel CES Booth

An experimental music sequencer which used a collection of physical cubes, each with a one-inch, 128x128 screen, and the ability to sense its neighboring devices. One cube kept the master sequence, and the others were individual instruments. This was a launch app for the Sifteo platform.

making music with one-inch screens

LoopLoop for Sifteo Cubes

Microsoft commissioned an animated visualization of <a href="https://www.kexp.org">KEXP's</a> playlist data and album art going back for years. Certain interactions could be automated, with the frame rate recorded and used to generate a GPU benchmark.

early GPU-powered web graphics

Radio Archive Visualization

The second-generation Surface table shipped with one application, Bing. It was meant to be used in public spaces and could route visitors to a destination on a map, or provide image search results in a novel way. Multiple users could perform tasks on the same display simultaneously. The source code was eventually released and some of the UI elements were granted patents.

multi-user image search and maps

Bing Search for Surface

An application which aggregates content from RSS, Twitter, and Flickr, and displays those items as an endless stream of news. Items can be pulled from the stream for closer examination and saved to a mobile device. A companion application customizes the look and content of the application for various contexts.

multi-user tweets, photos, and news

Social Stream for Surface

A touch table application which displayed stories about Microsoft’s global philanthropic efforts on a world map. The application uses a novel “lens” based UI to give multiple people the ability to explore stories and data visualizations simultaneously.

a zooming lens into a world map

Local Impact Map

An in-store experience for a mobile network which let customers place mobile devices on a table to learn their details and specifications. When two devices were placed near each other, relevant specs could be compared side-by side. Info about devices as well as service plans were stored in a custom CMS and deployed to all stores nightly.

play music and compare devices

WIND Mobile Retail

An experimental sequencer for the original Microsoft Surface table, where up to four users can make music together on four instruments distributed across four octaves, all playing to a master tempo. Released as freeware and exhibited at several events.

multi-user musical instrument

TouchTones

An application which used low-level Windows graphics APIs to display a multi-touch application switcher. The tool was used to allow meeting participants to queue up content and literally throw applications, websites, and other documents onto one of three massive shared screens, controlled by a Cisco CTS3000 Telepresence Suite.

multi-application screen sharing

SAP InSite Studio

An interactive video experience which annotates lectures with images, links, and transcripts. Students can save their progress through the videos as well as take time-synced notes within the application. The project launched with Richard Feynman’s Messenger Series lectures, a cornerstone set of seven talks at Cornell University in 1964.

study tools for Feynman's lectures

Project Tuva

Examples of the output of Kodak industrial printers were placed on an interactive table, detected, and surrounded with more information about how they were produced.

object recognition on the show floor

Kodak Product Explorer

An experiment in using mobile displays as lenses into the content on a larger fixed display. When a mobile phone is placed on the table top, it's detected and sent an image representing an alternative view of the content displayed beneath it. Multiple simultaneous devices are supported.

synchronized mobile displays

XRay

An original <a href="https://en.wikipedia.org/wiki/Microsoft_PixelSense">Surface table</a> was modified to sit on top of a <a href="https://en.wikipedia.org/wiki/Wii_Balance_Board">Wii Balance Board</a> and connected via a reverse-engineered protocol. Weight on the edges of the table was detected by the scale and added forces to a physics simulation on the display.

weird, tangible physics

Balance Bubbles

I spent time at Microsoft, working on the as-yet-unannounced <a href="https://en.wikipedia.org/wiki/Microsoft_PixelSense">Microsoft Surface touch table</a> product. I was in the Surface design studio and, without an official SDK, developed a number of high-polish proof of concept apps envisioning retail and restaurant scenarios which were presented by Bill Gates on the Today Show and Steve Ballmer at the D5 conference in 2007.

Projects