What Is Sheet Music Display in XR Piano? How It Works and Recommended Apps for Beginners [Meta Quest 3 Compatible]
Tomo
XR Piano
What Is Video Overlay in XR Piano?
Tomo
For those wondering, “Can you use XR piano without your real-world view being blocked?”
In this article, we’ll explain how “video overlay” — the core technology behind XR piano — works, and just how comfortable an experience it brings to playing piano.
Contents
What Is “Video Overlay” in XR?
Among XR technologies, the biggest defining feature of AR (Augmented Reality) and MR (Mixed Reality) is the ability to display digital information layered on top of your view of the real world, without losing that real-world view. Because virtual information is overlaid on top of the real-world footage, your performance on the real piano and the virtual guide blend together naturally.
Sigure Uses an MR (Mixed Reality) Approach
Sigure uses an MR (Mixed Reality) approach built on the Meta Quest 3 / 3s’s passthrough feature. Since only the note guide is overlaid while you continue to see the real keyboard, you get a natural experience while still using a real piano.
How the Meta Quest 3 / 3s’s Passthrough Makes Video Overlay Possible
The Meta Quest 3 / 3s features “full-color passthrough,” which captures real-world footage in real time using cameras on the front of the headset and projects it onto the lenses.
By overlaying Sigure’s note guide on top of this footage, the XR piano experience becomes possible.
Passthrough resolution and comfort
The Meta Quest 3 achieves full-color, high-resolution passthrough, letting you clearly see your real keyboard and your own hands.
The previous model, the Meta Quest 2, only offered monochrome passthrough, which made it hard to distinguish the white and black keys, sometimes getting in the way of playing. The Quest 3 has dramatically improved on this issue.
The note guide blends naturally with the real keyboard
By overlaying the note guide on top of the passthrough footage, you get the visual experience of “light falling onto your real keyboard.”
Since the virtual guide and your real hands appear within the same field of view, you can naturally press the keys while watching the guide.
Hands-On Explanation by the Sigure Development Team
When using Sigure on the Meta Quest 3, your piano and the notes overlap through crisp, clear passthrough.
4 Benefits Video Overlay Brings to Piano Playing
MERIT 01
You can see your real keyboard and hands
Your own hands and the keyboard remain fully visible. Being able to visually confirm exactly which key you’re pressing makes a huge difference in how reassuring it feels while you play.
MERIT 02
The real sensation matches the virtual information
Because the physical sensation of pressing a real key visually lines up with the virtual note guide, you get a natural playing experience where sound, touch, and sight all work together.
MERIT 03
You can play while staying aware of your surroundings
Since passthrough lets you see what’s around you, you can move around your room with confidence. You can even grab a drink or chat with family while keeping the headset on.
MERIT 04
Less eye strain
Since the visuals are based on real-world light, they tend to put less strain on your eyes compared to pure VR. This makes it easier to avoid eye fatigue even during long practice sessions.
Tips for Maximizing Video Overlay Accuracy
To get the most out of the overlay between the passthrough footage and the note guide, keep the following points in mind.
- Calibrate accurately: Calibration — which aligns the note guide with the real keyboard’s position — is the single most important step in determining video overlay accuracy. Be sure to redo this every time you restart the app.
- Wear the headset correctly: If the headset shifts position, your viewpoint shifts too, causing the passthrough footage and the note guide to fall out of alignment. Secure the strap firmly so it doesn’t move while you play.
- Set up an appropriate practice environment: In a space that’s too dark, the quality of the passthrough footage degrades. Make sure there’s enough light to clearly see your hands and the keyboard.
Frequently Asked Questions (FAQ)
- Q. Does the passthrough footage look as clear as reality?
-
A. The Meta Quest 3’s passthrough is full color and high resolution, providing a level of sharpness close to real life. That said, there’s a slight amount of latency and a small difference in resolution compared to your naked eye. The quality is more than sufficient for playing.
Q
- Q. Does the note guide appear perfectly aligned with the real keyboard?
-
A. If calibration is done accurately, it will appear aligned with almost no drift. You’ll need to recalibrate every time you restart the app.
Q
- Q. Is Sigure XR, AR, or MR?
-
A. Sigure uses an MR (Mixed Reality) approach built on passthrough. Since it overlays a virtual note guide on top of your view of the real keyboard, the experience is closer to AR (Augmented Reality). XR is the umbrella term covering all of these technologies.
Q
- Q. Can I use this in a dark room?
-
A. If it’s too dark, the quality of the passthrough footage degrades, making it harder to see your hands and the keyboard. We recommend using it in an environment with enough light to clearly see your hands.
Q
Try an XR piano experience that lets you keep your real-world view.
A Message from the Sigure Development Team
We believe the essence of XR piano lives in the simple experience of “playing along with a virtual guide while looking at your real keyboard.” The moment reality and the virtual world blend together naturally is one that always surprises first-time users.
We hope you’ll experience that feeling for yourself, even just once.
ABOUT ME
子どもの頃に挫折したピアノを大人になってから再開したとき、最初にぶつかったのが「楽譜が読めない」という壁でした。
「もっと気軽に、楽しく練習できる方法はないか」と考え続けた結果、XRピアノゲーム『Sigure』を作ることになりました。
このブログでは、同じように大人からピアノを始めた方・再開した方に向けて、XR・ARを使った新しい練習スタイルを発信しています。難しい音楽理論よりも「まず1曲弾ける」を大切に。一緒に楽しみながら上達しましょう。
このブログでは、同じように大人からピアノを始めた方・再開した方に向けて、XR・ARを使った新しい練習スタイルを発信しています。難しい音楽理論よりも「まず1曲弾ける」を大切に。一緒に楽しみながら上達しましょう。
