Partial Differentiation of Sensibilities and its Applications
- https://doi.org/10.2991/jrnal.k.201215.010How to use a DOI?
- Partial differentiation of sensibility, Tactile Score, T-bit, vibrotactile, SHOKKAKU
Sensibility information generally has various elements. To extract the time evolution of each component, we propose a partial differentiation of sensibility using the concept of partial differentiation. By this method, various sensibility information can be discretized and informatized. In this paper, we define the partial differentiation of sensibility and report case studies using this method.
- © 2020 The Authors. Published by Atlantis Press B.V.
- Open Access
- This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).
1. SENSIBILITY AS A DYNAMICAL SYSTEM
Our sensibility is a dynamical system in multi-dimensional sensory space, where each dimension corresponds to the sort of sensibility such as visual, audio, tactile, and so on. Music is the time change of sound, and the movie is the time change of vision. Tactile changes over time have not considered so far, but we have proposed a Tactile Score that represents tactile changes over time. On the other hand, the visual-spatial changes in a painting can also be transformed into temporal changes. When measuring spatial change, spatial movement occurs with or without the amount of sensitivity from point p1 to p2. In this paper, the path from p1 to p2 is ignored, and the focus is on the speed of travel. The movement speed is fast when glancing at the painting. On the other hand, if we want to appreciate the picture carefully, the movement speed is slow. Therefore, even spatial differences are described as temporal changes in sensibility.
Our sensory systems are composed of differentiators, and we perceive sensory stimulations through the partial differentiating sensory stimulations.
In order to calculate the partial differentiation of sensibility, we must set a measurable quantity of the sensibility. We can use a not only quantitative measure such as brightness of the picture, loudness of sound but also can use a quality measure such as the value of Semantic Differential (SD) method for measuring impressions or emotion such as beautiful, fear, happy.
The most crucial concept of the partial differentiation of sensibility measures the differences in time or/and space of sensibility; it shares the concept of partial differentiation in mathematics, but this is not a mathematical concept, and a mathematical equation of sensibility is not required; in calculating partial differentiation of sensibility, we use differentiable measure.
In the following, the notation of partial differential is borrowed. The reason is that sensibility is generally multivariable, and we discuss its temporal changes. Therefore, the temporal change of sensibility and the partial differential have similar conceptual meanings. We emphasize that the notation of partial derivatives in this paper does not claim that the amount of sensitivity is linear or continuous, not that it is a mathematical partial derivative.
2. THE PARTIAL DIFFERENTIATION OF SENSIBILITY
In general, sensitivity consists of cross-talks of multiple sensibilities. For example, music is affected not only by the pitch described in a music score but also by the time variation of timbre and volume. Here, the duration is the duration of the same pitch sound.
In case, we define that music is represented by four variables of pitch, duration, timbre, and volume, music and M have time changes of four sensibilities;
2.1. An Example of Partial Differentiation of Sensibility - Music
We derive the partial differentiation of sensibility, about the time change of the volume at the beginning of Beethoven’s Symphony No. 5. Figure 1 shows the result of calculation.
Next, we show an example of a partial differentiation about the pitch of the same music in Figure 2.
Not limited to this example, in the case of music, the partial differentiation of volume is the change in sound pressure that we call the waveform. And the partial differentiation for pitch becomes a musical score.
3. TACTILE SCORE
Sensitivity can be differentiated, as shown in the previous section. Obtaining differentiation is the discretization, and the discretization enables sensibility to compute digitally.
For example, a cheerful hello and a not-cheerful hello have the same character information. However, the sensibility of these two greetings is different. This difference is called the difference in wording. The characterization of “wording” has been done by frequency . We characterize the “wording” by the sensitivity partial differentiation concerning volume.
We show an example, the audio samples of saying “oha yoh gozai masu (good morning in Japanese)” as “cheerful” and “not-cheerful”. In this example, partial derivative of sensitivity, the time bin of the audio data was 0.01 s, and the maximum and minimum volumes were divided into 36 containers.
To explain the result of partial differentiation, we use a staff notation, where the pitch indicates loudness. Weak sound pressure is described as high sound, and strong sound pressure is described as low sound. The sound data was converted to musical instruments digital interface format and written in staff notation using score generation software (Figure 3).
The discretized result of the partial differential of sensitivity is called the tactile bit (T bit), and a staff notation describes T-bit is Tactile Score . Tactile Score was proposed by Suzuki et al. to represent time-varying somatosensory stimuli such as massages.
4. MEDIA TRANSFORMATION SYSTEM BY USING T-BIT AND TACTILE SCORE
Partial differentiation of sensibility can be used to rationalize media conversion while maintaining sensibility. We converted the dancer’s body movements into vibrotactile (Figure 4).
There are many elements in dance, but only body movements are partially distinguished. Only the speed of movement of the body is extracted from the dance. And all other elements are ignored.
We used OpenPose  for the motion capture of the body movements of three-dimensional coordinates of three points: wrist, ankle, and elbow. The speed and acceleration of the center of gravity position of wrist, ankle and elbow are discretized into five levels and converted into Tactile Scores. Then, the amplitude of vibration changed according to the Tactile Score. We call this system as Media Transformation System (MTS).
4.1. Case Report 1 – SHOKKAKU 2019
SHOKKAKU is a Japanese word, which means tactile sense. SHOKKAKU 2019 was an event on science and arts relating to tactile sense and held at Kyoto University Graduate School of Human and Environmental Studies (Figure 5).
We used MTS and transform body-movements of dance into tactile sense and created a performance artwork. This artwork was produced through collaboration with a Butoh dancer, Norihito Ishii (San-Kai-Jyuku Company) and a composer Eric Maestri.
We used a C960 webcam (eMeet Co., Ltd.) to shot live video for motion capture, video data send to PC (MacBook Pro), and OpenPose analyzed motion on MAX MSP (Cycling’74 Co., Ltd.). MTS generated a tactile score and transformed to sound data. By using an audio mixer, Eric added sound effect, then sound source for vibration was generated. The vibration sound source was distributed as vibration tactile sensation from four thunder sheets (made by Kolberg, 100 × 200 cm) arranged around the audience and 40 Haptic Vibration Displays (HVDs), worn by the audience (Figure 6).
We have originally developed the HVD to display vibration haptics through multiple sound exciters; a sound exciter (Dayton Audio Co., Ltd.) is an oscillator that responds to frequencies below the mid-range of the input audio signal. Thunder Sheet is a percussion instrument used to play the sound of thunder, as used in Richard Strauss’s Alpine Symphony. A Thunder sheet is usually played by hitting with a mallet, but we attached a sound exciter to the Thunder Sheet and vibrated. In the venue, four Thunder sheets were surrounded the audience and every audience wore a HVD. Hence, every audience had tactile sense transformed from dance’s movement directly from a HVD and in-directly from Thunder sheets.
4.2. Case Report 2 – Braille Tactile Score for Deaf-blind People
Deaf-blind people have multiple visual and hearing impairments and communicate via tactile sign language. Tactile sign language is a tactile representation of sign language used by the hearing impaired. Therefore, it is challenging to convey sensibility information. Hence, we propose Braille Tactile Score, which is the Tactile Score expressed as a vector of elements of sensibility. You can define every element of sensitivity, such as tactile sensation, strength, and contact area. For example, the tactile value can “comfortable (1)–unpleasant (5)”, the strength is “weak (1)–strong (5)”, and the contact area is “narrow (1)–wide (5)”; for example “Very comfortable, moderately strong, medium contact area” is (5,3,3).
N. Kawasora (deaf-blind), presented a poem using Braille Tactile Score at SHOKKAKU 2019. The created Braille tactile score was converted into a vibrating tactile sensation and distributed by Thunder Sheets and HVD.
5. FINAL REMARK
Sensibility information is converted to T-bit by partial differentiation of sensibility. T-bit makes it easier to handle sensitivity information, which was previously difficult to drive in a unified manner. Information technology in the 20th century has specialized in logic and data processing, but T-bit allows sensibility information to be integrated with conventional information technology.
CONFLICTS OF INTEREST
The author declares no conflicts of interest.
This research supported by SPIRITZ, Kyoto University and the author would like to thank, for the executive producer of SHOKKAKU 2019, Professor Masaaki Sakagami (Kyoto University).
Dr. Yasuhiro Suzuki
He is an Associate Professor of Graduate School of Informatics, Nagoya University, Japan. He graduated from Japan Advanced Institute of Science and Technology in 1995, received Doctor degree of Informatics from Kyoto University in 2001.
Cite this article
TY - JOUR AU - Yasuhiro Suzuki PY - 2020 DA - 2020/12 TI - Partial Differentiation of Sensibilities and its Applications JO - Journal of Robotics, Networking and Artificial Life SP - 258 EP - 261 VL - 7 IS - 4 SN - 2352-6386 UR - https://doi.org/10.2991/jrnal.k.201215.010 DO - https://doi.org/10.2991/jrnal.k.201215.010 ID - Suzuki2020 ER -