human computer interaction Tag

ASAP Journal

Happy to announce that my short article on machine learning, ASMR and sound “Automation as Echo” written with Sofian Audry is now published in ASAP/Journal 4.2 in a collection of articles assembled by Jennifer Rhee covering automation from diverse/creative/critical perspectives.

From the article:

“The echo is a metaphor that goes beyond sound, speaking to the physical and temporal gaps in human-computer interaction that open up a space of aesthetic consumption problematized by the impossibility of comprehending machine perspectives on human terms. The echo unfolds in time, but most importantly it unfolds in space: sound travels as a physical interaction between a subject and an object that seemingly “speaks back.”

The mythological nymph Echo “speaks” or “performs” her subjectivity through reflection or imitation of the voice of human Narcissus. Her (incomplete, sometimes humorous, sometimes uncannily resemblant) nonhuman voice is dependent on the human subject, who is also the progenitor of her speech. The relationship between these two mythological entities creates an apt metaphor for machine learning: its processes are not of the human, yet its “neural” functions are crafted in imitation of and in response to human thought. As machine subjectivity is crafted from human subjectivity, we cannot grasp its machined voice, nor perceive its subjective position, through analysis of its various textual, sonic, visual, and robotic outputs alone. Rather, the “voice” of machine learning is fleeting, heard through the spaces, the gaps, the movements between the machine and the human, the vibrational color of nonhuman noise.”

ABOUT ASAP JOURNAL

ASAP/Journal is a peer-reviewed scholarly journal published by John Hopkins University Press that explores new developments in post-1960s visual, media, literary, and performance arts. The scholarly publication of ASAP: The Association for the Study of the Arts of the Present, ASAP/Journal promotes intellectual exchange between artists and critics across the arts and humanities. The journal publishes methodologically cutting-edge, conceptually adventurous, and historically nuanced research about the arts of the present.

Elektra Festival Montreal

Project H.E.A.R.T. (Holographic Empathy Attack Robotics Team) (2017), my popstar/militainment VR game mashup with an affective control interface made in collaboration with Alex M Lee, is going to be featured at the upcoming Elektra festival XX in Montreal!  Look for it this June!

 

Exhibition opening: June 7, 6pm-9pm

Exhibition Dates: June 7-15, 2019

Perte de Signal 5445 Avenue de Gaspé local #107, Montréal, QC H2T 3B2, Canada

 

Click here to check out the full programming of Elektra Festival 2019

Book: Robotic Imaginary

My robotic artwork Swarming Emotional Pianos is featured in image and text on p 131-132 of Jennifer Rhee’s newly published book: The Robotic Imaginary: The Human and the Price of Dehumanized Labor (2018, University of Minnesota Press).  The image above is just a photo of me relaxing with a coffee as I read the first few pages…

This amazing book details AI from a perspective that is driven by emotion and humanity, while referencing the work and the influence of women and poc in a way I haven’t seen before. I found myself constantly thinking: yes, yes as I read the book!

 

From the official description of the book:

The word robot—introduced in Karel Čapek’s 1920 play R.U.R.—derives from rabota, the Czech word for servitude or forced labor. A century later, the play’s dystopian themes of dehumanization and exploited labor are being played out in factories, workplaces, and battlefields. In The Robotic Imaginary, Jennifer Rhee traces the provocative and productive connections of contemporary robots in technology, film, art, and literature. Centered around the twinned processes of anthropomorphization and dehumanization, she analyzes the coevolution of cultural and technological robots and artificial intelligence, arguing that it is through the conceptualization of the human and, more important, the dehumanized that these multiple spheres affect and transform each other.

Drawing on the writings of Alan Turing, Sara Ahmed, and Arlie Russell Hochschild; such films and novels as Her and The Stepford Wives; technologies like Kismet (the pioneering “emotional robot”); and contemporary drone art, this book explores anthropomorphic paradigms in robot design and imagery in ways that often challenge the very grounds on which those paradigms operate in robotics labs and industry. From disembodied, conversational AI and its entanglement with care labor; embodied mobile robots as they intersect with domestic labor; emotional robots impacting affective labor; and armed military drones and artistic responses to drone warfare, The Robotic Imaginary ultimately reveals how the human is made knowable through the design of and discourse on humanoid robots that are, paradoxically, dehumanized.

 

Click here to view more information on the book at University of Minnesota Press

 

esse magazine spring 2018

My work Swarming Emotional Pianos is featured in an essay written by Lindsay Leblanc in esse magazine 93 – Printemps / été 2018. The essay also features two media artists that I respect very much, David Rokeby and Jean-Pierre Gauthier.

Following is an abstract:

Sketchy Machines: Propositions Around Three Robotic Artworks

With the increasing presence of machines in public and private life, we continue to find new ways of articulating our relationships with them. In this article, the author uses the sketch as a frame of analysis for machine artworks by Canadian artists David Rokeby, Jean-Pierre Gauthier, and Erin Gee, and argues that the sketch is a fundamentally interdisciplinary and material state that accounts for machines’ complex engagements with human and other-than-human agents. Highlighting the sketch’s unfinished, imperfect, and adaptable qualities, the author attempts to define a “sketchy materiality” as it occurs in robotic art.
Lindsay LeBlanc

To learn more, or to purchase this magazine as a digital PDF  –
https://esse.ca/en/sketch

Algorithms that Matter @ IEM Graz

I’ve been selected to be a featured artist in residence at the Institut für Elektronische Musik und Akustik (IEM) in Graz, Austria, participating in the Algorithms that Matter Residency from April-June 2018.

From the ALMAT website:

“Algorithms that Matter is an artistic research project by Hanns Holger Rutz and David Pirrò.  It aims at understanding the increasing influence of algorithms, translating them into aesthetic positions in sound, building a new perspective on algorithm agency by subjecting the realm of algorithms to experimentation.

Almat is grounded in the idea that algorithms are agents that co-determine the boundary between an artistic machine or “apparatus” and the object produced through this machine. The central question is: How do algorithmic processes emerge and structure the praxis of experimental computer music? The hypothesis is that these processes, instead of being separated from the composer—as generators and transformers of infinite shapes—exhibit a specific force that retroacts and changes the very praxis of composition and performance.”

 

I will use this opportunity to extend my reach into exciting new forms of embodied algorithmicity, developing new techniques for combining physiological markers of emotion with algorithmic agencies.

To learn more about the research and proceedings of this residency, check out scans of my sketchbook, and transcriptions of conversations between myself and the other residents/researchers at IEM, click here to access our open exposition on the Research Catalogue online platform.

 

We acknowledge the support of the Canada Council for the Arts, which last year invested $153 million to bring the arts to Canadians throughout the country.

Nous remercions le Conseil des arts du Canada de son soutien. L’an dernier, le Conseil a investi 153 millions de dollars pour mettre de l’art dans la vie des Canadiennes et des Canadiens de tout le pays.

Ideas Lab Denmark

I will be giving a unique and in-depth workshop hosted by Emotional Data Lab (Aarhus University), Interactive Denmark and Ideas Lab in Aarhus, Denmark from November 21-23.  The workshop consists of 3 three-hour sessions where I will share my materials and experiences with incorporating physiological markers of emotion into the VR-compatible Unity environment.

Participants will be placed into “teams” in order to work together, experiment, and discuss the promises, problems and potential of using biosensors to capture a user’s emotional experience through digital tools.

Project H.E.A.R.T.

Project H.E.A.R.T. (2017)

2017

A biodata-driven VR game where militainment and pop music fuel a new form of emotional drone warfare.

A twist on popular “militainment” shooter video games, Project H.E.A.R.T. invites the viewer to place their fingers on a custom biodata device, and summon their enthusiasm to engage their avatar, Yowane Haku, in “combat therapy.” Fans of the Vocaloid characters may recognize Haku as the “bad copy” of Japanese pop celebrity Hatsune Miku, a holographic personnage that invites her fans to pour their content and songs into her virtual voice.

The biosensing system features a pulse sensor, and a skin conductance sensor of Gee’s design. Through principles of emotional physiology and affective computing, the device gathers data relative to heart rate and blood flow from index finger, and skin conductance from middle and ring fingers of users. The biodata is read by a microcontroller and transferred to Unity VR, thus facilitating emotional interactivity: a user’s enthusiasm (spikes in signal amplitude in skin conductance, elevated heart rate, and shifts in amplitude of the pulse signal) stimulates the holographic pop star to sing in the virtual warzone, thus inspiring military fighters to continue the war, and create more enemy casualties. At the end of the experience the user is confronted with their “score” of traumatized soldiers vs enemies killed, with no information whether this means that they won or lost the “game”.

The user is thus challenged to navigate soldier’s emotional anxieties and summon their positivity to activate Haku’s singing voice as soldiers battle not only against a group of enemies, but also against their own lack of confidence in times of global economic instability.

The landscape of Project H.E.A.R.T. was built from geopolitically resonant sites found on Google Maps, creating a dreamlike background for the warzone. In-game dialogue wavers between self-righteous soldier banter typical of video games, and self-help, bringing the VR participant to an interrogation of their own emotional body in a virtual space that conflates war, pop music, drone technology, and perhaps movement-induced VR nausea.

As Kathryn Hamilton pointed out in her 2017 essay “Voyeur Realism” for New Inquiry,

“VR’s genesis and development is in the military, where it has been used to train soldiers in “battle readiness,” a euphemism for: methods to overcome the innate human resistance to firing at another human being. In the last few years, VR’s usage has shifted 180 degrees from a technology used to train soldiers for war, to one that claims to “amplify” the voices afflicted by war, and to affect “world influencers” who might be able to stop said wars.”

Credits

Narrative Design: Sofian Audry, Roxanne Baril-Bédard, Erin Gee
3D Art: Alex Lee and Marlon Kroll
Animation and Rigging: Nicklas Kenyon and Alex Lee
VFX: Anthony Damiani, Erin Gee, Nicklas Kenyon
Programming: Sofian Audry, Erin Gee, Nicklas Kenyon, Jacob Morin
AI Design: Sofian Audry
Sound Design: Erin Gee, Austin Haughton, Ben Hinckley, Ben Leavitt, Nicolas Ow
BioSensor Hardware Design: Erin Gee and Martin Peach
BioSensor Case Design: Grégory Perrin
BioSensor Hardware Programming: Thomas Ouellet Fredericks, Erin Gee, Martin Peach
Featuring music by Lazerblade, Night Chaser and Austin Haughton
Yowane Haku character designed by CAFFEIN
Yowane Haku Cyber model originally created by SEGA for Hatsune Miku: Project DIVA 2nd (2010)
Project H.E.A.R.T. also features the vocal acting talents of Erin Gee, Danny Gold, Alex Lee, Ben McCarthy, Gregory Muszkie, James O’Calloghan, and Henry Adam Svec.

Thanks to the support of the Canada Council for the Arts and AMD Radeon, this project was commissioned by Trinity Square Video for the exhibition Worldbuilding, curated by John G Hampton and Maiko Tanaka.

This project would have not been possible without the logistical and technical support of the following organizations:

Technoculture Art and Games Lab (Concordia University)

Concordia University

ASAP Media Services (University of Maine)

Exhibition history

November-December 2017  Worldbuilding @ Trinity Square Video, Toronto

February-March 2018 Future Perfect @ Hygienic Gallery, New London Connecticut

April 26-28, 2018 @ Digifest, Toronto

June 7-17, 2019 @ Elektra Festival, Montreal

January 2020 @ The Artist Project, Toronto

October 2020 @ Festival LEV Matadero, Spain

Links

Project H.E.A.R.T. official website
Worldbuilding Exhibition Website
Review in Canadian Art
My research blog: Pop and Militainment
Featured on Radiance VR

Video

Project H.E.A.R.T (2017)
Installation and Gameplay

Gallery

VR Commission Update

Here’s a sneak peek at some of the art developed last summer in a residency at the Technoculture Art and Games lab at Concordia University with lead 3D artist Alex Lee, AI designer Sofian Audry, art assistant Marlon Kroll, and research assistant Roxanne Baril-Bédard. Among holographic popstars who may or may not have their own consciousness to begin with, the project includes rhetorical analysis of post 9/11 counterterrorist video games, reality television, startup culture, and self-help manuals for improving emotional state.

I am implementing the Biosensor control system this Winter and plan on working on finalizing the game’s art, music and sounds this summer for a launch towards the end of 2017 in an exhibition at Trinity Square Video in Toronto.


In the future, weapons of war possess advanced AI systems, systems that guarantee successful automated combat on behalf of soldiers wielding the technology.  The military still trains its soldiers in case of equipment failure, but at this point, fighters function more as passive operators. The terrorist threat has nothing similar to this technology in their ranks, and the effectiveness of our systems is swift and deadly.  Historically, our soldiers manning the machines have never witnessed violence or devastation at this scale: the largest threat to soldiers today defending our nation’s values is Post Traumatic Stress Syndrome.

To address this unfortunate state of affairs, the military developed a startup fund open to the public to resolve this issue through technological innovation.  Significant scholarships and research funding was provided for researchers interested in devoting time to creating a means towards mitigating the psychological crisis.  A risky but intriguing proof of concept was eventually presented: the creation of a revolutionary entertainment for the troops as they fought the terrorist threat.

Yowane Haku became the face of this entertainment: a mobile holographic pop star engineered specifically for psychological distraction on the battlefield.  

The world’s most talented engineers, design consultants, and pop writing teams were assembled to enshrine Haku with every aesthetic and technical element to impress not only the troops, but the world with her next-generation technology.  However, the initial test-run of this mobile holographic pop medium in combat trials was….a failure.  

On the battlefield, Haku’s perfect body glowed faintly amongst the dust and screams, bullets and explosions passing ineffectually, dance moves precise, vocalizations on point. But ultimately her pop music performance lacked resonance with the battle.  Instead of the soldiers being emboldened by this new entertainment, which was intended to distract or inspire them from their gruesome tasks, their adverse psychological symptoms…flourished.  Some of the men went mad, laughing maniacally in tune with the holographic presence smiling sweetly at them.  It was only due to the superiority of our AI weaponry and automated drone operation that the morally corrupt foreign threat, with their violent and technologically crude methods, were stopped that day. The minds of our soldiers were lost.

Months later, a young pool of startup professionals would provide another solution.  This vocal minority of engineers…though others called them crazy….had a hunch. For the hologram pop star to “work,” her systems needed access pure emotion, to link a human element with the trauma of the human soldiers.  But it was not clear who, or what, could best provide this emotional link…and what amount of embodied “disruption” this might entail…

This enthusiastically crowdfunded group of millennials completed their groundbreaking research without the strings of ethics funding or institutional control.  Human emotions and consciousness now flow direct to Haku via experimental trials in VR technology.  Haku rises again on the battlefront.

Simultaneously, a new reality television show has been borne of these first human trials. The star of this reality show could be…….you.

Could you be the next American Sweetheart?  Do you have what it takes to provide 110% Best Emotional Performance?  Join us through advanced VR technologies, Live and Direct on the battlefield, to find out if you could be fit to man the ultimate weapon of war: Our Next Holographic Idol.

This project is supported by the Canada Council for the Arts and Trinity Square Video’s AMD VR lab

Musicworks #126 Interview

Click here to read my interview with Alex Varty.  “ERIN GEE SINGS THE BODY ELECTRONIC”

Fresh on the heels of my return from the premiere of Echo Grey in Vancouver (my newest composition for vocal quartet, feedback soloist and tape), I find I’ve received my physical copy of Musicworks, which is a triannually released publication featuring experimental sounds from across Canada.

Amidst a really massive transition phase right now, I find that teaching full time has really changed what I can do as an artist.  Pushing myself to learn entirely new skillsets in organization and pedagogical performance (sidenote: yes, everything is a performance) has left me with little time or energy to invest in building new technologies.

Music composition has been something that I can invest time into, as all I need is a few moments, a microphone, my laptop, a notepad with pencil scribbles, my imagination.

This interview with Musicworks magazine was very interesting for me, as recently my opportunities have been coming from music composition.  The whole issue is actually very interesting, with a full feature on music and sound revolution in VR spaces, as well as some features on other very energetic and productive electroacoustic artists.

Musicworks #126 is available now with a special curated cd of sounds included in the physical magazine.  On this CD you can find a track from my Voice of Echo (2011) series.

New VR artwork commission from Trinity Square Video

I’m thrilled to announce that Trinity Square Video will be presenting new artworks for Virtual Reality interfaces in 2016-2017, including a new commissioned work by myself!  The work will feature pop music’s potential military applications in a first-person shooter style video game – expect autotuned voices, virtual pop stars, and new embodiments of my emotional biosensor hardware to take shape in this new work.

The project will feature Alex M. Lee as head artistic designer as well as work by Marlon Kroll and Roxanne Baril-Bédard.  I’ll continue to post teasers, hardware updates and more through this summer 2016!