Journal of Robotics, Networking and Artificial Life

Volume 6, Issue 3, December 2019, Pages 162 - 166

Evaluation for the Synchronization of the Parade with OpenPose

Authors
Yohei Okugawa, Masao Kubo, Hiroshi Sato, Bui Duc Viet
Department of Computer Science, National Defense Academy of Japan, Yokosuka, Kanagawa 239-8686, Japan
*Corresponding author. Email: em57037@nda.ac.jp
Received 9 November 2018, Accepted 30 November 2018, Available Online 16 December 2019.
DOI
10.2991/jrnal.k.191203.001How to use a DOI?
Keywords
OpenPose; unity; parade; synchronization; multi-person pose estimation; multi-agent
Abstract

In military organization, the synchronization in the parade is important for showing their majesty. However, we do not have any quantitative evaluation methods for the parade. In this research, as a first step of the quantitative evaluation for the parade, we propose an evaluation method focusing on the synchronization level of “Marking Time”, in which soldiers is moving their legs as in marching without step forward, between two persons with OpenPose. In order to measure the synchronization level, our method is based on the trend of the transition of arm swing angles in the parade, and the result shows our method is appreciable for measuring the synchronization level for two persons.

Copyright
© 2019 The Authors. Published by Atlantis Press SARL.
Open Access
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION

In the military parade, the synchronization is a major factor for the performance. Shimohagi, Ishihara, and Okugawa [13] conducted scientific analysis for the synchronization of the parade; however, it is difficult to measure the synchronization level of the parade because it requires a large number of posture data at same time. We do not have any quantitative evaluation methods for the synchronization level of the parade. In this research, we propose the evaluation method for the synchronization level with OpenPose. OpenPose [4] can measure many kinds of postures from images simultaneously, therefore, it is much easier to get a large number of posture data than conventional motion analysis with motion capture [5,6], and it is suitable for analyzing cooperative operation of multiple people such as parade (Figure 1).

Figure 1

Military Parade with OpenPose ( https://www.youtube.com/watch?v=zaviumhF7aA ).

As a first step of this research, for measuring the synchronization level of the whole of the parade, we focus on the arm swing of two persons in “Marking Time”. Marking Time is a kind of parade style, in which soldiers is moving their legs as in marching without step forward. In this paper, we analyze the synchronization level of the marking time, between two persons with OpenPose. When we evaluate the synchronization level of the two persons appropriately, we can evaluate the synchronization level of the parade.

This paper is organized as follows. After introducing OpenPose in the next section, we describe the data processing method with OpenPose. In Section 4, we propose the evaluation method for the synchronization level. After explaining the two experiments with 3D CG and real behavior of the marking time by two persons, we show our results from the experiments. In addition, this paper is updated with the results of reviews from the proceeding paper in ICAROB [7].

2. OpenPose

OpenPose represents simultaneous posture recognition of multiple people on single images. The left side of Figure 2 shows an example of pose estimation with OpenPose. If the movements of the two are synchronized, we can expect that we get the same trend data.

Figure 2

Multi-person pose estimation with OpenPose.

The right side of Figure 2 shows the points of posture data.

As the three-dimensional data of [x, y, likelihood] on each point, we can obtain 75 data for one piece of posture information. It is difficult to measure the synchronization because the values and the accuracy of the posture data are different depending on the position and body size. Therefore, it is necessary to use the trend of data which is not influenced by position or height as an index. We can get the posture data of each person in same frame with OpenPose, but the data is not related to the posture data in the before and the after frames. In order to obtain the time series posture data of each person, we did cluster analysis based on the location in each frame.

3. PROPOSED METHOD

In this section, we propose the measurement method using indicators focusing on inflection point of movement. In the following, we explain the proposed method in order.

3.1. Focus Points

Focusing on the synchronized arm swing of the left hand, we obtain the data of each shoulder (point 5 in Figure 3), elbow (point 6 in Figure 3) and palm (point 7 in Figure 3). Each data, which are points 5–7, are one-unit set. After we get time series data of each member, we converted the data into four angles: φ, ψ, λ, θ (Figure 3).

Figure 3

Focus points of OpenPose.

3.2. K-means Clustering

In order to get time series data of each person’s arm swing angle in the image, we conduct the clustering using K-means clustering [Equation (1)]. We use K-means clustering to the data of shoulder (5 in Figure 3), Given a set of data (x5frame1,x5frame2,x5frame3,,x5frame n) , K-means clustering aims to partition the n observations into k sets S = {S1, S2}, where μi is the mean of points in Si and Var(Si) is squared deviation of Si.

In this section, we analyze the marking time movies to get time series data of each two persons, therefore, we set K = 2 for the cluster analysis using K-means method.

argminSi=1kxSix-μi2=argminSi=1k|Si|Var(Si) (1)

3.3. Smoothing

In order to read the inflection point of the time series data of arm swing angle accurately, we use Simple moving average [Equation (2)] for smoothing out short-term fluctuations of the angle data. Simple moving average is the unweighted mean of the n data. When we have a series of the angle data (Angt, Angt+1, …, Angt+n), then the simple moving average Angsma is

Angsma=Angt+Angt+1++Angt+nn (2)

In this method, we set n = 8 for reading the inflection points accurately.

Figure 4 shows an outline of the data before and after the smoothing process. It can be seen that the outline is smooth, and the inflection points are easy to read.

Figure 4

Data smoothing before and after.

3.4. Evaluation

After smoothing out, we get the time series data of arm swing angle [Equation (3)].

A=[AngA1,AngA2,,AngAi] (3)
A is the time series data of arm swing angle of member A in an image, and AngA1 is the arm swing angle of member A at the first frame of the movie.

From each time series data of the arm swing angle, we use correlation coefficients for evaluation. We define the synchronization level r as follow [Equation (4)]:

r=i=1n(AngAi-AngA¯)(AngBi-AngB¯)((i=1n(AngAi-AngA¯)2)(i=1n(AngBi-AngB¯)2))12 (4)

We use Pearson’s moment correlation coefficient as the synchronization level. When we have the arithmetic mean AngA¯andAngB¯ , which is from the data A and B , the synchronization level r is expressed as [Equation (4)]. If r approaches 1, it is synchronized. If r approaches −1, it is expected that the phase will be in opposite.

4. EXPERIMENT

In this chapter, we conducted two kinds of experiments. In Experiment 1, we examined the measurement accuracy and the evaluation index of our method using 3D CG models. After confirming the effectiveness of the evaluation method, in Experiment 2, we evaluate the synchronization level r in real marking time behavior with our method.

4.1. Experiment 1: Accuracy of Our Method

For measuring the effectiveness of our method, we analyzed the 3D CG models created by Unity. Since we can adjust the synchronization level between two models, it can be an ideal analysis target. 3D CG is appropriate for evaluating the synchronization level of the parade because we can completely control human-like model.

In the 3D CG, the two models are only doing the marking time (Figure 5).

Figure 5

An example of 3D CG models with OpenPose.

The arm swing angle of the 3D CG models was implemented according to the simple vibration [Equation (5)]. In this section, the phase difference of the arm swing angle is expressed by changing the initial condition of θ0 as 0, π/8, π/4, π/2 and π.

x=Asin(ωt+θ0)(ω=2π) (5)

Figure 6 shows the synchronization level r of time series angle data of two models with four different angles. Each angle is represented different figure such as φ is represented as square. As the initial value θ_0 shifts from 0 to π, the value r decreases in four angles. Also, when the initial value θ_0 is π, that is, each arm swing angle is in opposite phase, it approaches −1 in four angles. The synchronization level of four angles are almost same in Figure 6.

Figure 6

The synchronization level r of 3D CG.

From the result, the synchronization level r is monotonically decreasing according to the value θ0. Therefore, it was confirmed that our evaluation method is appreciable for measuring the synchronization level in the 3D CG marking time movie.

4.2. Experiment 2: Evaluation of Marking Time

Since we can show our evaluation method is appreciable, we evaluated the synchronization of the real marking time. As in Experiment 1, we analyzed the marking time of two persons. We experimented with 6 pairs of two cadets in the National Defense Academy, which have parade training every day (Figure 7).

Figure 7

A capture of real marking time with OpenPose.

For Experiment 2, we have three types of arm swing: Phase Synchronization, Phase Difference and Phase Opposition. Figure 7 shows a snapshot of Phase Opposition. Both cadet arm swing is opposite in Figure 7.

Figure 8 shows the average of the synchronization level r of marking time. As shown in Figure 8, we can confirm that the synchronization level r decreases as the difference of the arm swing angle shifts.

Figure 8

The synchronization level r of marking time.

Figure 9 shows the synchronization level r in each angle with standard deviation. In Phase Difference case, we did not set the different level of the arm swing quantitatively like as Experiment 1. Therefore, the standard deviation of Phase Difference is wider than other cases. However, the results of Experiment 2 show the same trend as the result of Experiment 1, and it is our evaluation method is appreciable for measuring the synchronization level in the real Parade. In addition, it was suggested that measuring the synchronization level with armpit angle (φ) has good sensitivity for evaluation comparing to other angles.

Figure 9

The synchronization level r in each angle with standard deviation.

5. CONCLUSION

We propose the evaluation method for the synchronization level of the marking time with OpenPose. We focus on the arm swing angle and analyze the marking time for two persons. Through the experiment using 3D CG, it was confirmed that our method based on time series arm swing angle is appreciable for measuring the synchronization level. Moreover, through the experiment of real marking time of six pairs of two cadets, our evaluation method is appreciable for real environment. In addition, it was suggested that measuring the synchronization level with armpit angle has good sensitivity for evaluation.

Authors Introduction

Mr. Yohei Okugawa

He is the LTJG of Japan Maritime Self Defense Force. He currently belongs to the Department of Computer Science at the National Defense Academy of Japan as a graduate student. He graduated from the College of Media Arts, Science and Technology at the University of Tsukuba in 2014.

Dr. Masao Kubo

He is an Associate Professor of Dep. of Computer Science at National Defense Academy in Japan. He graduated from the Precision Engineering Department, Hokkaido University, in 1991. He received his PhD degree in Computer Science from Hokkaido University in 1996.

Dr. Hiroshi Sato

He graduated Doctor course at Dep. of Computational intelligence and System Science in Tokyo Institute of Technology in Japan. He is an Associate Professor of Department of Computer Science at National Defense Academy in Japan. His research interests include SICE, IEEJ, IEICE, JSAI member.

Mr. Bui Duc Viet

He received his bachelor’s degree from the Department of Computer Science, National Defense Academy, Japan in 2019. He is currently a Master course student in National Defense Academy, Japan.

REFERENCES

[1]T Shimohagi, Parade analysis based on network system, in Proceedings of National Defense Academy of Japan Graduation Thesis (2017), pp. 61-62. (in Japanese).
[2]T Ishihara, Development of VR parade training system, in Proceedings of National Defense Academy of Japan Graduation Thesis (2018), pp. 47-48. (in Japanese).
[3]Y Okugawa, M Kubo, T Shimohagi, T Ishihara, and H Sato, Analysis of a parade with the Kuramoto model -for better performance without trainings-, in Proceedings of IEEE SMC, IEEE (Miyazaki, Japan, 2018), pp. 4005-4010.
[5]K Inada, T Ishikawa, and H Watanabe, Coincidence evaluation of multiple people’s dance using OpenPose, in Proceedings of the 80th National Convention of IPSJ, IPSJ (Tokyo, Japan, 2018), pp. 257-258.
Journal
Journal of Robotics, Networking and Artificial Life
Volume-Issue
6 - 3
Pages
162 - 166
Publication Date
2019/12/16
ISSN (Online)
2352-6386
ISSN (Print)
2405-9021
DOI
10.2991/jrnal.k.191203.001How to use a DOI?
Copyright
© 2019 The Authors. Published by Atlantis Press SARL.
Open Access
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

TY  - JOUR
AU  - Yohei Okugawa
AU  - Masao Kubo
AU  - Hiroshi Sato
AU  - Bui Duc Viet
PY  - 2019
DA  - 2019/12/16
TI  - Evaluation for the Synchronization of the Parade with OpenPose
JO  - Journal of Robotics, Networking and Artificial Life
SP  - 162
EP  - 166
VL  - 6
IS  - 3
SN  - 2352-6386
UR  - https://doi.org/10.2991/jrnal.k.191203.001
DO  - 10.2991/jrnal.k.191203.001
ID  - Okugawa2019
ER  -