Watch a demo video
1. Set up the MIDI Port on your computer
Windows
On Windows, the browser does not directly expose internal virtual MIDI ports. To send and receive MIDI messages, you need to install a virtual MIDI port. Some common solutions:
- loopMIDI – creates easy-to-use virtual MIDI ports: https://www.tobias-erichsen.de/software/loopmidi.html
- LoopBe1 – another lightweight virtual MIDI port: https://www.nerds.de/en/loopbe1.html
- VB-Cable / MIDI – complete virtual audio/MIDI package: https://www.vb-audio.com/Cable/
After installing the virtual MIDI port, launch it and connect it to the applications you want to use. The application will automatically detect the available MIDI port in the dropdown menu at the top left.
macOS
On macOS, no additional software installation is required. You can use the “Audio MIDI Setup” application to create virtual MIDI ports. Go to Applications → Utilities → Audio MIDI Setup, then add a new virtual MIDI port. It will automatically appear in the browser.
Linux
On Linux, you can use ALSA MIDI or JACK MIDI to create virtual ports:
- ALSA: use
aconnectto create ports and connect them between applications. - JACK: with
QjackCtlyou can easily manage virtual MIDI ports.
The application will detect available ALSA MIDI ports in the browser.
2. Permissions
To use the application, you need to authorize the browser to access the webcam and MIDI. Without these permissions, the application will not be able to detect hand movements or send MIDI messages. When prompted by the browser, allow access to the webcam and MIDI port.
3. MIDI port selection and webcam management
At the top left, there is a dropdown menu to select the MIDI port. The first available port is automatically selected and activated. Next to it is a button to show or hide the webcam: hiding the webcam can improve performance if the live video is not needed.
4. Left hand: playing notes and sending CC
- By moving the left hand over the grid on the left half of the screen, you can play up to three notes simultaneously (thumb, index, and pinky).
- Moving the hand closer or farther changes the range of notes covered.
- For each finger of the left hand (thumb, index, pinky), MIDI CC messages are sent based on the X and Y position of the finger.
5. Right hand: controlling the 3D cube
- By moving the right hand on the right half of the screen, you can move and rotate a 3D cube.
- The X and Y position of the cube and rotations along the three axes generate MIDI CC messages sent on channel 1.
6. MIDI Sliders
- At the bottom, there are 12 sliders that function as faders:
- Green sliders: controlled by bringing together the thumb and index of the left hand.
- Yellow sliders: controlled by bringing together the thumb and pinky of the left hand.
- By joining the thumb and index or pinky, you can move the sliders up and down and send MIDI CC messages.
- Each slider sends a MIDI CC message with values from 0 to 127 on channel 2.
- Below each slider is the corresponding MIDI CC number.
7. Head, eyes and mouth: send MIDI CC messages
- Moving your head in the three directions and opening and closing yout eyes and mouth, you will send MIDI CC messages on channel 3.
8. MIDI Message Monitoring
At the top right, you can view the list of all MIDI messages sent, showing the channel, controller, and value.
9. Online Test with Cardinal (VCV Rack 2 Clone)
Cardinal is a clone of VCV Rack 2. It allows you to simulate EuroRack modules and create patches with oscillators (VCO), filters (VCF), envelopes (ADSR), amplifiers (VCA), sequencers, arpeggiators, effects, and much more. You can control your patches using this application.
Instructions:
- You can test Cardinal without installing it, at this link: https://cardinal.kx.studio/live
- I created a demo patch that you can download by clicking here and then import into Cardinal via the File > Import Patch menu. The patch is ready to be used and you can start playing right away. In addition to hand movement, you can use the first two green sliders to adjust the volume of the first two instruments (pads), and the first two yellow sliders to adjust the volume of the other two instruments (percussion and FM). The third and fourth green sliders control other parameters of the FM instrument.
- Warning! You must click the Engine > Enable / Reconnect MIDI menu, otherwise Cardinal will not receive MIDI messages. You’ll also need to authorize MIDI usage when prompted by your browser.
- I also created a template patch to help you start building your own custom patches. You can download it by clicking here and then import it into Cardinal via the File > Import Patch menu.
- This patch already has MIDI input configured to receive 12 MIDI CC messages on channel 1 and 12 MIDI CC messages on channel 2 (in addition to notes). You can control up to 24 different parameters.
- The first two green sliders are already mapped to the faders of the first two mixer channels, and the first two yellow sliders are mapped to the faders of the third and fourth mixer channels.
- To map other parameters, right-click on one of the CC encoders, click Map 1 > Start Mapping, and then click on the target parameter you want to control (for example, one of the encoders on a Eurorack module).
- You can also adjust the range of each parameter by clicking Map 1 > Range and then dragging the range with your mouse.
- You can map up to 4 different parameters to each encoder.
- Important! When you’ve finished creating your patch, to use the MIDI interface and make music, you must return to the Hand Tracking MIDI application screen (the one with the webcam). Only when that screen is active can you send MIDI controls to other applications.
- Note! To use Cardinal online, you still need to follow the instructions in section 1. Set up the MIDI Port on your computer