A Tactile Sense Centered Virtual Reality game by using Biometric feedback
- DOI
- 10.2991/jrnal.k.200909.003How to use a DOI?
- Keywords
- Virtual reality; game; tactile sense; bio-feedback
- Abstract
We construct a Virtual Reality, VR system centered on the tactile sensation. Tactile sensation has a more significant effect on sensibility than audiovisual. Therefore, if tactile sensation and audiovisual sense are combined, we can create “virtual presence” in the VR space. In this system, a user’s biological information is linked to the audiovisual data presented. A player plays with virtual objects through physical body movements. And the system feeds-back biometric information and deliver the change of biometric data through the sensory presentation to the user.
- Copyright
- © 2020 The Authors. Published by Atlantis Press B.V.
- Open Access
- This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).
1. INTRODUCTION
In conventional Virtual Reality (VR) technology and applications, visual information is mainly used [1], while auditory and tactile information have been used side by side. In this research, we construct a VR system that focuses on tactile sense and secondary to audiovisual perception. The tactile sensation has a strong influence on sensitivity as compared with the audiovisual sense [2]. For example, it has shown that vibrotactile plays a significant role in the perception of music, and by amplifying certain vibrations in a concert venue or music reproduction system can improve the music experience [3]. And it has reported that higher visual realism can also give more robust fear responses which may be indicative of stronger presence [4]. Therefore, by combining vibrotactile and visual stimuli, we aim to provide the virtual presence to the virtual object in the VR space. To do that we measure the user’s biometric information [5] and make a game controller tangible and create a virtual object in cooperation with audiovisual information. Through feedback, the sensory presentation via biometric information, virtual items to generate a sense that users are cooperating with virtual objects.
2. HEARTOID
The purpose of this research is to give a sense of presence to a virtual object in the VR space by feeding back the baptized biological information based on the idea of the “heart picnic [6].” Therefore, we created a virtual object in the VR space that made the user’s biometric information tangible and constructed a tactile VR game system, the “Heartoid [7]”, using the virtual object.
This system is a VR game that touches a virtual beating object (sphere) that makes the player’s heart sound tangible. This sphere corresponds to a vibratory tactile sensation presentation device in the heart picnic, and by touching this sphere existing in the VR space via the controller, the player can touch the tunable heart sound. The tactile presentation device used was a tactile presentation device built into the controller of HINE’s VINE VR. The sphere regularly vibrates finely based on the results of real-time measurement and analysis of the heartbeat sound and the movement (acceleration) of the player, and the amplitude of the vibration synchronized with the magnitude of the heart sound — the subtle vibration selected according to the player’s biological condition and movement.
The player’s body and the sphere are connected by a thin thread (Figure 1). The thread visually complements the tunable heart sound, suggesting that the player’s heart sound sent to the sphere, and suggesting a relationship between the player and the area with the thread. While the string is connected, the area floats in the air against gravity and is alive.
From the game screen. The user and the heart (sphere) are connected by wires. The user feels the heartbeat from the tactile feedback of heart sounds.
2.1. Flow of the Game
2.1.1. Introduction
The player touches the sphere and feels the heartbeat of participant’s heart through the vibration of the sphere.
2.1.2. Target game (about 3 min)
The goal is to hit the sphere and break 50 targets in time (Figure 2). The heart rate fluctuates during the game. Also, the larger the sphere, the easier it is to hit multiple targets at the same time. Therefore, it is necessary to make physical movements and make utterances. Therefore, vocalization and respiratory heart rate fluctuate. The change of the mental and physical state is presented by tactile sensation and made tangible.
Snapshot of the game. The black circle in the center is the sphere corresponding to the heart. The ring around it is the target. The user throws a sphere to break the target. Fifty targets generated.
2.1.3. Ending
The player cuts the thread connecting the sphere to himself. At the end of the game, the controller becomes scissors. Use the scissors to cut the thread. When the thread is reduced, the sphere drops according to gravity and stops vibrating. The ball can be grabbed and thrown but does not respond in response to the user’s heart rate. Also, throwing a sphere will not return to the player’s hand (Figure 3).
Ending of the game. When the target application game ends, the controller becomes scissors. The user cuts the sphere connected to himself with the scissors using the scissors. When the thread reduced, the sphere falls by gravity. The sphere does not beat and does not return to me when thrown.
3. SYSTEM COMPONENTS
We used the HTC Vive for the VR head-mounted display, controller, and tracking system. For the tactile presentation, we used the tactile presentation device built into the controller unit attached to Vive.
The player’s heart sound was acquired by using a stethoscope with built-in microphone (Yagami Co., Ltd., Nagoya, Japan, heart sound microphone attached to the experimental heart sound meter HBS-NA) and connected to a PC with a stereo mini-plug. A 500 Hz low-pass filter was used to reduce microphone noise and improve heart sound detection performance.
3.1. Realtime Bio-feedback
3.1.1. Analyzing body movements
To implement the whole system, we used the Unity, which is a platform to create video game by Unity Technology Inc.
In Unity, the velocity and angular velocity of the controller are saved as a three-dimensional vector and dynamically updated within a fixed frame of physics calculation every 0.01 s. Hence, to obtain the acceleration, the difference between the velocity and angular velocity vector in the previous frame and the current velocity and angular velocity vector is taken and divided by the time (0.01) between the frames to obtain the acceleration and angular acceleration vector.
By taking the magnitude of these acceleration and angular acceleration vectors, a three-dimensional vector is collected into a scalar. After that, one scaled data is obtained by multiplying the volume of acceleration and the magnitude of the angular acceleration of the same frame by applying appropriate correction terms. The general expression of the nth frame (1), where Vn denotes Controller’s velocity vector at the nth frame, angVn: Controller’s angular velocity vector at ∆t denotes time between frames k and l, that are arbitrary 32-bit floating-point number, while V0 = (0, 0.0).
The system obtains time-series data of acceleration, by adding this data to the end of the array over 20–30 physics calculation frames until the tactile analysis is completed. Since the input array (one-dimensional time series data) is cleared once for each analysis, the acceleration data at the previous examination does not affect the current analysis result.
3.1.2. Analyzing biometric data
A developed application server for biometric information analysis is set up, and POST, which is a request method supported by Hyper Text Transfer Protocol (HTTP); HTTP loads hypertext link in the World Wide Web (WWW). POST delivers binary time series data; The server analyzes biometric information based on the request and returns the result to the client-side as an HTTP response.
HTTP communication of this system is implemented by WWW class which is Unity HTTP communication wrapper class. Since HTTP communication by WWW class is executed by the coroutine function, which is an asynchronous process in Unity, the game simulation execution loop does not stop during biometric information analysis.
However, there is a risk that the execution frame rate will be reduced due to the overhead of converting a vast numerical array into a byte string for a request. For this, the transformation was divided into several frames and divided. To confirm its effectiveness, we requested data to a local application server and measured the average time taken for a response to be about 0.3 s. Therefore, it is possible to analyze the tactile sensation biometric information about every 0.3 s while the game is running.
Note that this Web application analysis method enables the development of applications with real-time biometric information analysis functions in various development environments, as long as it is an environment that supports HTTP communication.
In other words, not only Unity, but also Unreal Engine, which has a proven track record for higher-end applications, and Cocos-2dx, which has a proven track record in 2D games, HTTP communication is supported as standard, so this real-time tactile analysis is possible. There is high versatility.
Also, when building an application development environment such as a game engine for in-house use, this real-time tactile analysis can be easily performed by supporting HTTP communication and creating a wrapper.
3.1.3. Tactileization
This system uses vibrotactiles to perform tactileization. For tactileization of biometric information, it is necessary to select the vibrotactile sensation that is output according to the analysis result of biometric data. In this system, the biological information was classified, and the vibrotactile sense assigned to each cluster in advance was presented.
For the selected vibrotactile, the output voice block was analyzed for each execution frame, the value of vibration intensity was determined for each block, and it was output to the vibration of a device such as a controller. The reason why one value is obtained at the time of vibration output is that the vibration Application Programming Interface (API) in devices such as game controllers can often output only one value of vibration strength (driving motor driving force).
4. CONCLUSION
In this research, we demonstrated the possibility of developing a haptic content game that affects sensibility by measuring and analyzing biological information and presenting haptics in real-time (making haptic). A user evaluation test will be performed in the future. Still, a user who tried for the preliminary survey said, “The sphere (corresponding to the heart) is alive because the changes due to physical movement are fed back by tactile feedback. Some comments have been foreseen that it affects sensibility, such as “I get a feeling.” And “I hesitate to cut the wire at the end.”
As future tasks, in addition to user evaluation, improvement of the execution speed of biological information analysis in this system will be significant for broader application. At present, it is not possible to support game designs that require review at every game frame (less than 0.0032 s for 30 FPS). Therefore, by speeding up the processing, it is desired to improve the method so that the technique can be applied to a game application requiring high-speed processing such as a shooting game.
CONFLICTS OF INTEREST
The author declares no conflicts of interest.
ACKNOWLEDGMENT
The author would like to thank Mr. Yoshihito Ushida, for his sophisticated skills and outstanding efforts. This research was supported by the
AUTHOR INTRODUCTION
Dr. Yasuhiro Suzuki
He is an Associate Professor of Graduate School of Informatics, Nagoya University Japan. graduated from Japan Advanced Institute of Science and Technology in 1995, received Doctor degree of Informatics from Kyoto University in 2001.
REFERENCES
Cite this article
TY - JOUR AU - Yasuhiro Suzuki PY - 2020 DA - 2020/09/11 TI - A Tactile Sense Centered Virtual Reality game by using Biometric feedback JO - Journal of Robotics, Networking and Artificial Life SP - 156 EP - 159 VL - 7 IS - 3 SN - 2352-6386 UR - https://doi.org/10.2991/jrnal.k.200909.003 DO - 10.2991/jrnal.k.200909.003 ID - Suzuki2020 ER -