How Does XR Piano’s Keyboard Guide Work? A Clear Explanation of the Technology Behind Light Overlaying the Keys [Meta Quest 3 Compatible]
For those curious about the mechanics behind it: “How does the XR piano keyboard guide actually display light on top of a real keyboard?”
In this article, we’ll clearly explain the technology behind XR piano’s keyboard guide, without relying on difficult technical jargon.
- What Is XR Piano’s Keyboard Guide?
- The 3 Technologies Behind the Keyboard Guide
- Display Patterns for the Keyboard Guide
- The Process Behind How Sigure’s Notes Appear on the Meta Quest 3
- Factors That Affect the Keyboard Guide’s Accuracy
- Frequently Asked Questions (FAQ)
- A Message from the Sigure Development Team
What Is XR Piano’s Keyboard Guide?
The XR piano keyboard guide is a feature where, as you play piano while wearing a headset like the Meta Quest 3, light or color is displayed directly over the real keyboard to indicate “which key to press next.” By leveraging XR technology (including AR and MR), the real keyboard and the virtual guide blend together naturally.
There’s no need to read sheet music — you can play a song simply by following the light cues and pressing the indicated keys. Many people with no prior piano experience have reported being able to play songs like “Sakura Sakura” or “Twinkle Twinkle Little Star” within just a few minutes.
The keyboard guide is displayed as a color overlay directly on the keys. As a note falls from above and crosses the line of a key, that key lights up — so go ahead and play it.
The 3 Technologies Behind the Keyboard Guide
XR piano’s keyboard guide is made possible by combining the following three technologies.
Passthrough (capturing the real-world view)
The Meta Quest 3 features “color passthrough,” which captures real-world footage in real time using cameras on the front of the headset and projects it onto the lenses. This lets you clearly see your hands and the keyboard even while wearing the headset. This is made possible through the use of MR (Mixed Reality) technology.
The previous model, the Meta Quest 2, only offered monochrome passthrough, but the Quest 3 upgraded this to full color and higher resolution — which is what allows the real keyboard and the virtual guide to overlap and appear natural together.
Spatial projection (displaying notes at the keyboard’s position)
Windows and notes can be projected three-dimensionally into the visible space around you.
Once the keyboard’s position has been registered, the guide will continue to display correctly over the keys even if you move your head while playing. (This is reset when you close the app.)
Accuracy detection
Song data is managed in MIDI format. MIDI is a digital file format that records performance information — which note to play, when, and at what intensity.
By comparing the song’s MIDI data against the MIDI data generated by the notes you actually play on the piano, the system can judge your performance in real time as Great, Good, or Bad.
By combining passthrough, spatial projection, and MIDI-based accuracy detection, you get the experience unique to XR piano: a guide that lights up in real time directly on top of your real keyboard.
Display Patterns for the Keyboard Guide
The way the keyboard guide is displayed varies by app, but there are mainly two common patterns.
Falling-note style
A display style, similar to rhythm games, where notes fall from above toward the keyboard. The timing is visually easy to understand, making this style especially popular with beginners.
Light-up key style
A style where the actual key you need to press lights up. Since the guide is integrated directly into the keyboard itself, it’s intuitively clear which key to play.
The Process Behind How Sigure’s Notes Appear on the Meta Quest 3
In Sigure, the keyboard guide is displayed through the following process.
Use a USB-MIDI cable to directly connect your digital piano to the Meta Quest 3. This allows your keystroke data to be sent to the app in real time.
Use the in-app calibration feature to have your headset memorize the position of your real keyboard. The more accurately you complete this step, the less the guide will drift out of alignment.
Once you choose the song you want to practice, notes will start falling over the keyboard. Press the keys in time with the notes, and your score increases based on Bad/Good/Great ratings and combo count.
Use the auto-accompaniment function and the Stay Notes function to repeatedly practice just the difficult sections. We recommend playing slowly while watching the guide, then gradually increasing the tempo.
We filmed the entire process from launching the app to playing, in one continuous take. Touching the on-screen menus in mid-air might feel a bit tricky at first, but it becomes smooth and natural once you get used to it.
Factors That Affect the Keyboard Guide’s Accuracy
If you find the keyboard guide misaligned or hard to see, the following factors are usually the cause.
- Calibration accuracy: Accurately registering the keyboard’s position during the initial setup is the single most important step in preventing guide misalignment. You’ll need to recalibrate each time you restart the app.
- How well the headset fits: If the headset shifts position, your viewpoint will shift too, making the guide appear out of alignment with reality. Be sure to fasten the head strap securely.
Frequently Asked Questions (FAQ)
- Q. Does the guide line up perfectly with the real keyboard?
-
A. If calibration is done accurately, the guide will line up with almost no misalignment. However, you’ll need to recalibrate if you restart the app or if the headset shifts position.
- Q. Can I use the guide without a digital piano?
-
A. You can display a virtual keyboard on a table or desk and use just the guide, but virtual keyboards in this kind of virtual space have poor responsiveness, which reduces the practice benefit. We recommend connecting a digital piano via MIDI when using Sigure.
- Q. Can I change the color or speed of the guide?
-
A. In Sigure, you can customize the color of the notes. You can’t change the scroll speed directly, but you can adjust the tempo of the song itself. We recommend practicing while adjusting these settings to fit your level.
- Q. Can the guide keep up with fast phrases?
-
A. Yes, it displays without any issues. Songs with a large number of notes may take a bit longer to load, but there’s no delay once the song starts playing.
If you have a Meta Quest 3, you can start practicing with the keyboard guide right now.
A Message from the Sigure Development Team
Once you understand how the keyboard guide works, we think it’ll click for you why it’s so easy to play with. Through the careful coordination of three technologies — passthrough, spatial projection, and MIDI-based detection — you get the feeling that the keyboard itself is lighting up to teach you what to play.
We developed Sigure with the goal of delivering that experience in the simplest way possible, so that anyone can use it. We kept the difficult setup to a minimum, and maximized the joy of playing. We hope you’ll give it a try for yourself.
このブログでは、同じように大人からピアノを始めた方・再開した方に向けて、XR・ARを使った新しい練習スタイルを発信しています。難しい音楽理論よりも「まず1曲弾ける」を大切に。一緒に楽しみながら上達しましょう。