Technology Research – Capturing the Contemporary Conductor

Capture Day!

After months of planning and two days of setting up equipment, Capture Day finally took place on 27th September 2017. The team arrived early to make sure the systems were up and running ready for the conductors and ensemble.

Three conductors – our own Ben Oliver, Holly Mathieson and Geoffrey Paterson were captured in turn, each having rehearsal time with the ensemble prior to giving two performances which were both captured.

Thanks to Vicon’s generosity in lending us additional cameras and devices for the capture system, as well as technical support to optimise the system, we managed to obtain high quality face, body and hand data from the conductors.

Conductor Holly Mathieson being marked up by Dan Halford

In addition to the Vicon motion data, we have video of the conductors and ensemble from 7 cameras (including a 360 degree video) in various locations, plus motion data from a Kinect 2, audio recording from stereo and close microphones and muscle data from four wireless EMG devices on the conductors’ biceps and triceps.

Conductor Geoff Paterson with markers and EMG sensors on view.

The day ran very smoothly, with help from Harry Matthews, Beth Walker and Sergiu Rusu (student and recent graduates) to help with unsticking and sticking markers amongst other activities!

Richard Polfreman and Dan Halford keeping an eye on the tech!

We are deeply grateful to Bob Dimmock, Phil Bacon, Matt Oughton and everyone at Vicon for all their help and generosity in making this day a success. We’d also like to thank Geoff and Holly for being such willing guinea pigs, and also our ensemble, for their wonderful patience and playing: Anna Durance (oboe + electronics); Vicky Wright (bass clarinet); Julian Poore (trumpet); Joley Cragg (percussion); Liga Korne (electric piano); Aisha Orazbayeva (violin) and Dan Molloy (double bass).

Finally, of course our thanks to The British Academy and the Leverhulme Trust for their funding of the project.

A Visit from Vicon

This week we were especially pleased to welcome a team from Vicon Motion Systems to the University. Vicon are a world-leading developer of motion-capture systems and they came to show us their face capture technologies to explore their potential use in our Capturing the Contemporary Conductor research project – Richard Polfreman (Music), Benjamin Oliver (Music), Cheryl Metcalf (Health Sciences), and research assistant Dan Halford. Vicon mo-cap systems are used in biomechanics research as well as the entertainment industry, where facial expression capture is used to drive animated characters to bring them more realistically to life. Previous studies of conducting have highlighted the importance of facial expressions in communicating to the ensemble, and so we are interested in capturing this information in addition to body motion and beyond simple video recording.

Matt Oughton (EMEA Sales Manager), Dami Phillips (Technical Sales Engineer) and Katie Davies (Support Engineer) arrived at the motion capture lab with two boxes of kit which we were keen to unpack and try out as soon as possible. First was their brand new Cara Lite system, recently announced at SIGGRAPH 2017. This two-camera based system can be used markerless with analysis software to model the movement of the wearer’s face once recorded. The system certainly felt lighter than the previous Cara system, and this version is designed to be customisable to different client needs.

Research team member, composer and conductor Dr Benjamin Oliver wearing the Cara Lite motion capture system.

Next we tried the original Cara system, a complete 4-camera true 3D marker-based solution providing complete 3D motion data from the markers on the subject’s face. The marker layout is up to the user, and so this can be customised for the particular level of detail needed.

Engineer Katie Davies helping Research Assistant Dan Halford try on the Cara face capture system.

The Cara headset fitted very snugly and felt secure and comfortable. Heavier than the Cara Lite, but the openness of the camera mounts facilitated a clear line of sight to the score for conducting. The fact that the end result of the processed capture is the 3D motion data of markers (as the body motion data is) may be helpful.

Dr Benjamin Oliver practicing with the Cara headset

Depending on the application, the main Vicon system can be used without a headset to capture facial expressions, but whether this is appropriate depends on the range movement of the subject, the number of cameras in the system, the other markers being captured etc. For the Capturing the Contemporary Conductor project, we are carrying out tests to decide which option will work best for us!

Our grateful thanks go to Matt Oughton and the team at Vicon for bringing their amazing systems to show us, it was a very interesting and helpful meeting.

How to capture conducting motion?

In recent months I’ve been looking at how conducting motion has been captured in the past, and the applications for which it has been used. Conducting motion has been explored in a number of different ways previously. While we are planning to use infra-red retro-reflective marker-based motion capture, it is useful to review work by others in the field, both for performance studies and for computational analysis and control.

In general, four main types of transducer have been used: optical, inertial, electromagnetic and bioelectrical, each of which has its own advantages and disadvantages. In general, bioelectrical can be discrete, but is an indirect measure of motion, subject to noise and other issues; electromagnetic detection can cover a large capture volume and has no line-of sight problems, but suffers from electromagnetic interference and historically relatively large transmitters; likewise inertial sensors can cover a large space with no line-of sight problems, but usually suffer from drift and is rather intrusive; optical systems range from simple video (with computational analysis) through to high-precision 3D retro-reflective or active marker IR tracking, all of which require clear lines-of-sight and can have other issues such as ease of portability and susceptibility to lighting issues.

Bioelectrical sensors have been used to read muscle activity from the bicep/tricep combination, notably Marrin’s Conductor’s Jacket used EMG sensors to provide arm movement data for control of other systems or for study.

Electromagnetic sensors can provide 3-axis position and orientation data from a small sensing object placed in a generated magnetic field. These were also used in one version of the Conductor’s Jacket, and also Ilmonen and Takala for gesture recognition. Max Matthews’ Radio Baton electromagnetic controller has also been used in conducting experiments.

Inertial sensors have been used in a number of studies, including using wiimotes (e.g. Bradshaw and Ng), as well as other MEMS devices in projects such as Augmented Conductor, mConduct, and Conducting Master. Inertial devices measure acceleration and orientation (and magnetic) data which can be used to calculate the position of the of the sensor as it is moved, although drift remains an issue with these.

Optical approaches vary from simple video capture and manual analysis, to baton mounted LED’s with computational video analysis, the use of 3D gaming controllers (Microsoft Kinect) and high-end optical mocap systems from the biomechanics and entertainment industries. In CtCC (this project!) we will be taking this final approach due to the potentially very high level of detail and precision achievable, but we may supplement this with data from other devices for comparison with other studies and assessment of the quality of other systems.

What does a conductor of contemporary music do?

Over the past few months I have been undertaking a literature review of books focused on conducting and music direction of contemporary music. This has included: conducting related pedagogical textbooks/manuals; books by and interviews with leading conductors; texts dealing with more general issues in performing contemporary music; and doctoral dissertations on issues relating to conducting.

The focus of this work has been on the physical aspects of conducting focusing on the musical intention of different gestures and approaches. There are, of course, numerous other facets to conducting including, for example, rehearsal techniques, personal and professional interaction with performers and score preparation. These aspects, and many more, have been given much attention in the literature by scholars and practicing conductors alike. There is also an emerging field of ‘conducting studies’ in which scholars have explored and written about the practice of conducting in terms of music semiotics, gesture studies and from anthropological standpoints. Here, however, the concentration is on the physical gestures involved in conducting with a specific focus on contemporary music practice.

The original intention was to create a taxonomy of conducting gestures but it has become apparent that this is likely an impossible (and doomed) task as there are so many different viewpoints, conducting techniques and schools of thought when it comes to thinking about the physical gestures required of a conductor. Numerous scholars and conductors have underlined the inherent individuality of different conductors and that conducting ‘technique’ is fluid and hard to pin down to a series of specific gestures.

However, there are a number of key gestures, basic techniques and conventions that conductors are taught, or develop through practice, that allow them to communicate effectively (or not) with performers. The approach in this review, therefore, has been to try and summarise some of the different ideas and approaches of conducting technique as a starting place for thinking about what motions it might be useful to ‘capture’ in the Capturing the Contemporary Conductor project. This review is also helping me focus my ideas for writing a new piece for the session with live musicians and guest conductors in September 2017.

IMU Based Motion Capture

Ben Oliver wearing the Neuron motion capture suit, with Dan Halford

While the final capture session for the project will focus on using highly accurate optical motion capture, as a preliminary test we wanted to quickly capture some conducting motion data for analysis.

We had access to a Perception Neuron IMU mocap suit, designed for small animation/game studios and education establishments. This uses put to 32 “Neurons” or small Inertial Measurement Units which house gyro, accelerometer and magnetometer. The Axis Neuron software translates this data into 3D position data for each Neuron and streams this in real-time and can record it to disk.

Willing victim Dr Benjamin Oliver was strapped into the Neuron suit during a rehearsal with Workers Union Ensemble which was also video and audio recorded so we could compare the data from each source.

We captured approximately 60 minutes of rehearsal time, which was then integrated by Research Assistant Dan Halford, a sample of which can be seen below which features Helen Papaioannou‘s Backscatter (2017).

Project Introduction

Capturing the Contemporary Conductor is a BA/Leverhulme Trust funded pilot-project investigating the use of high-quality motion capture systems for the study of conducting gesture. It is a collaborative project between Music and Health Sciences at the University of Southampton.

The research team comprises:

Dr Richard Polfreman (Music)

Dr Benjamin Oliver (Music)

Dr Cheryl Metcalf (Health Sciences)

and research assistant Daniel Halford.

For any enquiries related to the project, please contact Richard.