Gang LIU, Jing WANG

Institute of Robotics and Intelligent Systems, Xi’an Jiaotong University, Xi’an 710049, China

Abstract: There are two famous function decomposition methods in math: the Taylor series and the Fourier series.The Fourier series developed into the Fourier spectrum, which was applied to signal decomposition and analysis.However, because the Taylor series function cannot be solved without a definite functional expression, it has rarely been used in engineering. We developed a Taylor series using our proposed dendrite net (DD), constructed a relation spectrum, and applied it to decomposition and analysis of models and systems. Specifically, knowledge of the intuitive link between muscle activity and finger movement is vital for the design of commercial prosthetic hands that do not need user pre-training. However, this link has yet to be understood due to the complexity of the human hand. In this study, the relation spectrum was applied to analyze the muscle-finger system. One single muscle actuates multiple fingers, or multiple muscles actuate one single finger simultaneously. Thus, the research was focused on muscle synergy and muscle coupling for the hand. The main contributions are twofold:(1) The findings concerning the hand contribute to the design of prosthetic hands; (2) The relation spectrum makes the online model human-readable, which unifies online performance and offline results. Code is available at https://github.com/liugang1234567/Gang-neuron.

Key words: Taylor series; Relation spectrum; Dendrite net (DD); Prosthetic hands; Machine learning;Engineering

1 Introduction

Myoelectric prosthetic hands, where amputees control the prosthetic hand by voluntarily contracting their residual muscles, are attracting considerable critical attention (Zhuang et al., 2019). To our knowledge,they are classified into four types according to electromyography(EMG) decoding methods:(1)Type 1 is that two electrodes are attached to the residual muscles, and then the corresponding joint movement is actuated in proportion to the EMG amplitude (Farina et al., 2014). (2) The function where the user can switch the active joint by a cocontraction of both muscle groups or other heuristics is added to type 1 to produce type 2 (Parker et al., 2006). (3) To control more degrees of freedom(DOFs),intensive research is focused on motion classification that assigns EMG features to a discrete set of motions (Kuzborskij et al., 2012; Atzori et al., 2016). (4) Recently, an approach that maps muscle activation to force or motion by training a complex “black-box” neural network (NN) has been investigated for simultaneous and proportional myoelectric control (Ngeo et al., 2014; Zhuang et al.,2019). Without a doubt, type 1 is the most convenient and intuitive. Its intended function corresponds to the physiologically appropriate muscles.Today, most commercial devices use this method.However, these devices usually use a two-recordingchannel system to control a single DOF,because the intuitive link between the muscle activity and finger movement is unclear.

The key to the intuitive link is to parse muscle synergy and muscle coupling for the hand, and a white-box model of the muscle-finger system is the basis for the analysis. Jiang et al.(2009)presented a DOF-wise nonnegative matrix factorization (NMF)algorithm to extract the wrist’s neural control information from EMG.However,it has not yet been used for finger movement, probably because the hand is much more complicated than the wrist. In addition,it is worth emphasizing that although many machine learning models have been used for proportional nonlinear myoelectric control,they do not show the intuitive link due to their nature (Oskoei and Hu, 2007;Zhuang et al., 2019). An early study proved that mechanical coupling and muscle coupling limit finger independence (Lang and Schieber, 2004). Both mechanical coupling and muscle coupling are useful for the design of myoelectric prosthetic hands (Brown and Asada, 2007), but the muscle coupling of the finger has not been systematically investigated.

This paper presents the relation spectrum in engineering based on the dendrite net (DD), the first white-box machine learning algorithm(Liu and Wang, 2021). DD can be found in Liu and Wang(2021),who gave a proof-of-principle of similarity between the relation spectrum and Taylor series. From the Fourier-like perspective(Bracewell,1978),DD is akin to the Fourier transform and the relation spectrum is similar to the Fourier spectrum. However,the Fourier spectrum is the decomposition of the signal, and the relation spectrum is the decomposition of the model or system. For the specific engineering problem,this study aims to explore the intuitive link between the muscle activity and finger movement.The framework of this engineering problem is shown in Fig. 1. Also, this paper is the first application of DD.

2 Materials and methods

This section describes the relation spectrum in engineering based on DD by solving the specific engineering problem of the muscle-finger system.

Fig.1 Framework for solving the engineering problem of unclear intuitive link between the muscle activity and finger movement

2.1 Subjects

The data analyzed in this paper are from scientific intramuscular electromyography (iEMG) data related to the isometric hand muscle contractions of 14 subjects (Malesevic et al., 2020). These subjects were divided into two protocols: the first focused on the muscles available within a short residual forearm(SRL),and the second focused on fingers and thumb muscles(LRL).

The 12 subjects had six pairs of fine-wires inserted regardless of the protocol. However, another two subjects were recorded with nine electrodes whose positions met both SRL and LRL. Thus, the two subjects provided four groups of data and were regarded as four subjects. We obtained eight SRL subject data and eight LRL subject data. According to the preliminary test and the introduction in Malesevic et al.(2020),we selected the LRL subject data that focused on fingers and thumb muscles and named subjects 1-8. The LRL protocol targeted the following muscles: flexor digitorum profundus(FDP), extensor digitorum communis (EDC), abductor pollicis longus (APL), flexor pollicis longus(FPL, responsible for thumb flexion), extensor pollicis longus (EPL, responsible for thumb extension),and extensor indicis proprius (EIP, responsible for index finger extension)(Malesevic et al., 2020).

2.2 Acquisition setup and protocol

2.2.1 Acquisition setup

Multiple sensors were used to record hand forces and corresponding muscular activities during the experiments. Hand forces were measured using a custom-made force measurement device (Malesevic et al., 2020). The intramuscular EMG signals were recorded using the Quattrocento (OT Bioelettronica, Torino, Italy) biomedical amplifier system. All iEMG signals were sampled with 16-bit amplitude resolution at 10 240 Hz. A high-pass filter at 10 Hz and a low-pass filter at 4400 Hz were used during recordings. The intramuscular electrodes were paired fine-wire electrodes from Chalgren, Gilroy,USA.

The positioning of the fine-wire electrodes was performed by a medical specialist in clinical neurophysiology using the guidelines from theAnatomical Guide for the Electromyographer: the Limbs and Trunk(Perotto,2011;Malesevic et al.,2020).

2.2.2 Acquisition protocol(Malesevic et al.,2020)

Fig. 2 shows the acquisition protocol. The subject sat on a chair comfortably and was instructed to place the hand in the force measurement device.The whole measurement protocol was controlled and guided automatically by the custom-made software developed in Labview. The subject was asked to produce force/torque that matched the cue presented on the screen. Sinusoidal waveforms were provided as visual cues. The rationale behind the sinusoidal tracking task was to generate a gradual force increase and provide iEMG and force data that described the muscle-finger system. Concretely,the repetition frequency was set to 0.1 Hz to enable a gradual and controllable force increase.

2.3 Data processing

2.3.1 Pre-processing

The following steps were executed for all subjects. To obtain the iEMG data without 50 Hz noise and its harmonics, iEMG data were filtered with a 10 Hz high-pass filter,a 450 Hz low-pass filter,and a notch filter at 50 Hz (Kuzborskij et al., 2012). The root mean square(RMS)is one of the most common EMG signal features and represents the signal envelope. In this study, as in Malesevic et al. (2020),the RMS was calculated using a 250-ms-wide window (MATLAB command: square, smooth). Then we obtained the iEMG data and force data after removing obvious noise for further analysis.

Fig. 2 Acquisition protocol. Reprinted from Malesevic et al. (2020), Copyright 2020, with permission from the authors, licensed under CC BY 4.0

2.3.2 Muscle-finger system models

We should tune the model that we select to simulate the real system and then analyze it through the trained model for modeling and analyzing the muscle-finger system. Implementing these concerns requires a white-box algorithm that can resolve variable relationships between dependent variables and multiple explanatory variables. As far as we know,the current generalized white-box algorithm is only multiple regression. Traditional multiple regression is usually converted into linear regression through linear processing. Then, the typical least square method is used to obtain the preset parameters(Poggio, 1975; Schielzeth, 2010; Liu and Wang, 2021).Thus, multiple regression is essentially a linear regression. In the EMG interface, there are too many EMG electrodes, which leads to too many items and high computational complexity (Poggio, 1975;Schielzeth, 2010; Liu and Wang, 2021). Therefore,multiple regression generally contains only firstorder terms in the EMG interface according to Hahne et al. (2014). This paper compares DD with firstorder multiple regression,also known as multiple linear regression (LR). A 10-fold cross-validation (10-FCV) strategy was used for both LR and DD to evaluate the overall performance. Note that this paper focuses on the relation spectrum that“reads” the trained model of DD. For poorly performing simplified models (LR), this paper just compares the performance.

1. Linear regression

A linear regression is an algorithm that models the linear relationship (Kutner et al., 2005). For all subjects, we built the LR models about a single finger as follows:

wheref(x)represents the finger force of a single finger,E(t)represents the iEMG RMS of muscles,andAlrrepresents the regression coefficients.

2. Dendrite net

Various NNs have been employed to model the relationship of the input space and the output space(Hornik et al., 1989). However, traditional NNs,called cell body nets (Liu and Wang, 2021), are like a black-box and provide an unreadable model.

The dendrite net,which imitates biological dendrites in brains,is another novel basic machine learning algorithm(Liu and Wang,2021). Unlike machine learning algorithms that search for an appropriate classification curve or surface, DD aims to design the logical extractor with controllable precision and is a white-box algorithm with lower computational complexity.

Here,we built a DD model of the muscle-finger system using three modules for each subject. The overall DD architecture is shown in Fig. 3. The architecture can be represented by the following formula:

Fig. 3 Dendrite net architecture of the muscle–finger system. EFDP, EEDC, EAPL, EFPL, EEPL, and EEIP denote the iEMG RMS of FDP, EDC, APL, FPL,EPL, and EIP, respectively

2.4 Relation spectrum

2.4.1 Fourier series(Tolstov,2012)

Any functionf(x) can be decomposed into the sum of trigonometric functions:

These coefficients of the trigonometric polynomial form the Fourier spectrum. Later,the advent of the fast Fourier transform greatly extends our ability to implement the Fourier spectrum on digital computers (van Loan, 1992). Today, it has become an essential tool for decomposing signals.

2.4.2 Taylor series

Taylor series of a real- or complex-valued functionf(x)infinitely differentiable at a real or complex numberais the power series (Abramowitz and Stegun, 1972):

In the more compact sigma notation, this can be written as

wheref(n)(a) denotes thenthderivative offevaluated at pointa.

Suppose we have determinedmpoints off(x).The Taylor expansion at each point is as follows:

Then,f(x)can be expressed as including a Taylor expansion with all sample points:

Eq.(10)can be generalized to functions of more than one variable:

where (xd ?aid)nd,i ∈{1,2,···,m}can be simplified as the form that contains constant items and items containingxj,j ∈{1,2,···,d}. The expression in the generalized matrix form can be expressed as follows:

whereX=(x1,x2,···,xd), andWTaylorrepresents the coefficient matrix of polynomial. Note thatWTaylorcontains the derivatives at sample points in Eq. (11). These are similar to those in DD using backpropagation and the chain rule. The dendrite module is expressed as follows(Fig. 4):

whereAl-1andAlare the inputs and outputs of the module, respectively,Xdenotes the inputs of DD,andWl,l-1is the weight matrix from the (l ?1)thmodule to thelthmodule.

Fig. 4 Generalized dendrite module

Note thatXdenotes the inputs of DD. Thus,it is seen as a constant inderivation of the single DD module (Eq. (15)).

Eq.(15)can be expressed in the following form:

wheremdenotes the number of training samples in one batch,andαis the learning rate.

In backpropagation,every time through one DD module, the adjusted weight of the corresponding module is obtained by the first derivative. The order of the derivative contained in the weight of the module gradually increases according to the position of the module (from back to front).

In essence,DD can be expressed in the generalized form:

whereWDDrepresents the weight matrix (strength of synaptic connections). Note thatWDDcontains the derivatives at sample points in Eq. (11). Meanwhile,DD calculations contain only matrix multiplication and the Hadamard product. Therefore,WDDof the trained DD can be translated into the relation spectrum about inputs and outputs by formula simplification with software(e.g.,MATLAB or Python).The relation spectrum with a large number of coefficients can be expressed using a figure and a table,such as Fig. 8 and Table 1. The item index and coefficients are the abscissa and ordinate,respectively,and the items can be found in the table automatically generated by a computer. The relation spectrum expresses the effects of the inputs on outputs, and the impacts contain independent and interaction effects to different degrees. It may become an essential tool for decomposing a system or an online model. In fact, before this strategy, we had to use different models for offline analysis and online operation due to the black-box nature of online models, such as the traditional NN model or support vector machine(SVM) model. DD and relation spectrum can integrate offline analysis into the online operation or take an online model into offline analysis. Additionally, an orthogonal basis in signal decomposition of signal processing avoids repeated extraction of the same power. The result of DD is shown as “addition.” Therefore,no repeated extraction is present.

2.5 Muscle synergies for the hand

We can solve the relation spectrum by formula simplification with software for the muscle synergies mechanism on the fingers. In this study,we calculate each subject’s relation spectrum, which represents muscle synergies for a single finger and then determine the items with the same contribution. These items with the same contribution are useful in designing prosthetic hands with intended functions that correspond to the physiologically appropriate muscles. The proportion of items with the same contribution is defined as

wherenis the number of subjects, co(i) is the coefficient of the corresponding item of the model for

2.6 Muscle coupling for the hand

Because DD has been transformed into the relation spectrum,we can analyze the muscle coupling for a hand,which has not been done before. Combining the relation spectra of all subjects, we calculate the correlation coefficient of the relation spectrum between pairwise fingers. These correlation coeffi-cients represent the muscle coupling of five fingers and are calculated using the following formula (Devore,2011):

For clarity,a flow diagram of analysis using DD is shown in Fig. 5.

Fig. 5 Flow diagram of analysis using the dendrite net

3 Results

3.1 Muscle–finger system models

Fig. 6 shows that the DD models outperform the LR models in 10-FCV accuracy for the musclefinger system, in terms of bothR2and MSE. As an example,the results of subject 3 are shown in Fig.7.For thumb F-E, the data from subjects 2 and 4 are suspected outliers due to the electrode quality and are discarded(Malesevic et al.,2020)(R2: thumb FE: 0.771±0.108(LR)<0.827±0.084(DD); thumb A-A: 0.608±0.230 (LR)< 0.653±0.234 (DD); little: 0.777±0.064(LR)< 0.780±0.053(DD); ring:0.757±0.114 (LR)< 0.780±0.111 (DD); middle:0.505±0.292 (LR)< 0.618±0.237 (DD); index:0.734±0.067(LR)<0.770±0.079(DD)).

Fig. 6 The 10-FCV accuracy of LR and DD models: (a) R2; (b) MSE. P-values were calculated using paired samples t-tests. ?P < 0.05, ??P < 0.01, R2 or MSE: mean±SD. References to color refer to the online version of this figure

3.2 Muscle synergies for a single finger

Since the advent of the NN, it has been seen as a black box. For several decades,an NN that can explain the relationship between the input space and output space has been intensively sought.

The muscle-finger system cannot be analyzed intuitively due to its complex structure. This paper shows the relation spectrum of the muscle-finger system (Fig. 8). The relation spectrum can be read by way of a checklist. The item index is referred to Table 1. Despite some differences across subjects, a lot of the same-contribution items exist in subjects.

Fig. 7 Results of the DD model for the muscle–finger system. As an example, the top figure shows results for testing data of subject 3 in the 10-fold cross-validation, in which (1)–(6) represent thumb F-E, thumb A-A,little, ring, middle, and index, respectively. References to color refer to the online version of this figure

3.3 Muscle coupling for a single finger

The knowledge of muscle coupling for the hand is useful in the design of bionic prosthetic hands. Although the rough relationship between muscle and finger has been found through anatomy and physiology, there was no method for analyzing muscle coupling using online models prior to this study. This paper shows the muscle coupling of the human hand.Despite differentiated coupling strength across subjects due to evolution, the common muscle coupling is shown in Fig. 9. The results are interesting and can be verified using our own hands. Note that we move only the active finger in Fig.9 and keep others relaxed when verifying the muscle coupling.

Fig. 8 Relation spectrum of the muscle–finger system in DD models. The definition of the same-contribution proportion can be found in Section 2 (Eq. (19)). The same-contribution proportion: In the above eight subjects, the proportion is the number of positive/negative correlations (in the number of positive and negative correlations, the correlation with the largest number is regarded as the correlation of the item)divided by eight and then multiplied by 100%, which is the proportion of the item’s same contribution in all subjects. As an example, same-contribution items across subjects are marked. Refer to Table 1 for the index of the item. References to color refer to the online version of this figure

4 Discussion

4.1 Specific engineering: the muscle–finger system

As mentioned in the literature review, the knowledge of the intuitive link between the muscle activity and finger movement is conducive to the design of commercial prosthetic hands with no need of user pre-training. However,this link is unclear. The present study was designed to explore the intuitive link.

In terms of muscle synergies for a single finger,we present the relation spectrum of the muscle activity and finger movement. Some relations of the findings are consistent with those in Perotto (2011)and Malesevic et al.(2020). Nevertheless,our results

Fig. 9 Muscle coupling of the hand: (a) correlation coefficient; (b) visualization. The results are interesting and can be verified using our own hands: (1) Raise one hand according to the perspective in (b); (2) All fingers are relaxed; (3) Only the active finger is moved while the others remain relaxed. Note that we have tried our best to flex or extend to the maximum and feel the muscle coupling carefully. Due to the individual differences, the intensity of coupling may be different

Table 1 Items in the relation spectrum

were more precise. For example, previous research showed only that index finger movement has a positive correlation with activation of EIP. Our study showed similar results,but also revealed that the index finger movement is negatively correlated with the activation of FPL (Fig. 8 and Table 1). One unexpected finding was that the index finger movement is negatively correlated with the square of activation of EIP and positively correlated with the square of FPL activation. This suggested that the intensity of the same muscle activation may affect the corresponding finger’s movement direction. In addition, this showed that the relationship between muscle activation and finger movement is nonlinear,which explains why the DD models outperform the simplified LR models. Because this study is focused on the technology itself, the medical information is not discussed in more detail. Details can be found in Fig. 8 and Table 1. Additionally, because of the noise associated with the electrode character and environment,the difference in signal strength,and each individual’s specificity,there were some differences in the coefficient magnitude. Nevertheless,the relation spectrum is similar across subjects.

Prior studies have noted the coupling phenomenon of the hand (Wu et al., 2001; Lang and Schieber, 2004; Brown and Asada, 2007). The coupling of the hand can be divided into passive mechanical coupling and active muscle coupling. Passive mechanical coupling was measured using a measuring implement(Lang and Schieber,2004)and has been used to design bionic hands(Brown and Asada,2007). This study assessed active muscle coupling.These muscle coupling results agreed with those obtained in an earlier study(Lang and Schieber,2004).However, our results were more precise because our method provides a quantitative analysis of iEMG DD models instead of an analysis by which the indices obtained in the passive condition are subtracted from those obtained in the active condition (Lang and Schieber, 2004). Details can be found in Fig. 9.These were particularly useful for prosthetic hands.Specifically, using the results of muscle coupling of human fingers (Fig. 9), in future research we can design prosthetic hands that have a coupling relationship similar to human hands. Looking at Fig.8,for future research we can refer to muscle synergy results and imitate the muscle synergy of human hands to drive prosthetic hands with fewer power sources. Additionally,the muscle-finger system’s relation spectrum provides a reference for commercial myoelectric prosthetic hands with no need of user pre-training.

4.2 Relation spectrum

In our previous study,we presented the generalized DD (Liu and Wang, 2021). This paper demonstrated the similarity between the Taylor series and the generalized DD by proof-of-principle and presented the relation spectrum. The muscle-finger system’s logical relationship is relatively simple,and we do not need more dendrite modules. However, for a more complex model, we may use more dendrite modules. To solve the vanishing gradient problem,a residual strategy can be used (He et al., 2016). The residual dendrite (ResDD) module is expressed as follows(Fig. 10)(Liu, 2020):

whereAl-1andAlare the inputs and outputs of the module, respectively,Xdenotes the inputs of the residual DD, andWl,l-1is the weight matrix from the (l ?1)thmodule to thelthmodule.

Fig. 10 Residual dendrite module

whereWRDDrepresents the weight matrix(strength of synaptic connections). Note thatWRDDcontains the derivatives at sample points in Eq. (11). These derivatives in the residual DD using backpropagation and the chain rule are similar to those in the Taylor series. Meanwhile,the residual DD calculations contain only matrix multiplication,matrix addition,and the Hadamard product. Thus,WRDDof the trained ResDD can be translated into the relation spectrum of inputs and outputs by formula simplification with software(e.g.,MATLAB or Python).

Additionally,traditional machine learning algorithms (e.g., NNs, SVM, or decision tree) generate only a black-box model. Therefore, there are usually differences between the algorithms used in offline analysis and online applications. Sometimes,in traditional methods,offline analysis shows promising results but online analysis shows poor performance.According to our present experiments in the applications of the muscle-finger system, the proposed DD and relation spectrum showed potential advantages in analyzing the reasons for poor online performance. Further investigations and experiments will be conducted to apply them in other fields in the future. However, theoretical results show that the relation spectrum demonstrates the trained/online model using DD. The online model can be “read” in offline analysis,which unifies online performance and offline results.

Table 2 Comparison of typical algorithms

5 Conclusions

This paper demonstrated the similarity between the Taylor series and DD using backpropagation and the chain rule, and then presented a relation spectrum. It is widely known that a functional relationship can be expressed by the sum of some trigonometric or power items. For the expression of a trigonometric polynomial,the typical example is the Fourier frequency spectrum. Here, the relation spectrum is the spectrum of the power series. The relation spectrum expresses the impact of inputs on outputs,and the impacts contain independent and interaction effects to different degrees. The relation spectrum and DD unify online performance and offline results.

In terms of specific engineering, we have solved a significant problem-the unclear link between the muscle activity and finger movement-through DD and relation spectrum. Our contribution lies in the relation spectrum of the muscle-finger system and the knowledge of muscle coupling,which may provide a reference for commercial prosthetic hands.

Before concluding, note that this paper has shown the concept of the relation spectrum of DD from a proof-of-principle for the first time systematically, and the relation spectrum complemented DD in general engineering. Although this relation spectrum and DD,like SVM,are very basic,they may be applied in most engineering fields. One can imagine their impacts in the future (Table 2).

Contributors

Gang LIU presented the relation spectrum and drafted the paper. Jing WANG offered advice on this study. Gang LIU and Jing WANG revised and finalized the paper.

Compliance with ethics guidelines

Gang LIU and Jing WANG declare that they have no conflict of interest.