Shui Li,Zhungzhung Yn,Yixin Guo,Xioyn Su,Yngyng Co,Bofeng Jing,Fei Yng,Zhnguo Zhng,Dwei Xin,Qingshn Chen,Rongsheng Zhu,*

a College of Engineering,Northeast Agricultural University,Harbin 150030,Heilongjiang,China

b College of Arts and Sciences,Northeast Agricultural University,Harbin 150030,Heilongjiang,China

c College of Agriculture,Northeast Agricultural University,Harbin 150030,Heilongjiang,China

Keywords:Soybean Feature pyramid network PCA Instance segmentation Deep learning

ABSTRACT Mature soybean phenotyping is an important process in soybean breeding;however,the manual process is time-consuming and labor-intensive.Therefore,a novel approach that is rapid,accurate and highly precise is required to obtain the phenotypic data of soybean stems,pods and seeds.In this research,we propose a mature soybean phenotype measurement algorithm called Soybean Phenotype Measure-instance Segmentation (SPM-IS).SPM-IS is based on a feature pyramid network,Principal Component Analysis(PCA) and instance segmentation.We also propose a new method that uses PCA to locate and measure the length and width of a target object via image instance segmentation.After 60,000 iterations,the maximum mean Average Precision(mAP)of the mask and box was able to reach 95.7%.The correlation coefficients R2 of the manual measurement and SPM-IS measurement of the pod length,pod width,stem length,complete main stem length,seed length and seed width were 0.9755,0.9872,0.9692,0.9803,0.9656,and 0.9716,respectively.The correlation coefficients R2 of the manual counting and SPM-IS counting of pods,stems and seeds were 0.9733,0.9872,and 0.9851,respectively.The above results show that SPM-IS is a robust measurement and counting algorithm that can reduce labor intensity,improve efficiency and speed up the soybean breeding process.

1.Introduction

Soybeans are one of the world’s main staples,and are an important source of oil,protein and healthcare materials [1].Soybeans compose a very important portion of people’s food intake [2].In recent years,through the unremitting efforts of soybean breeders,soybean yield has been significantly improved [3].However,it is still necessary to develop and cultivate new soybean varieties with high a yield,high quality and multiple different resistances[4].The plant breeding process requires the phenotypic analysis of a large range of populations[5].These phenotypic studies are an essential link in the breeding process.The phenotype of the soybean main stem is the most direct factor that determines soybean strength.Strength is primarily indicated by lodging resistance,which is one of the important factors for reducing soybean yield and quality[6].The number of pods,seed size and number of seeds per pod directly affect the soybean yield,and the number of pods is positively correlated with the number of soybean seeds[7].Therefore,it is necessary to quickly obtain this phenotypic data in the form of phenotypic analysis indicators during the breeding process[8].

In order for soybean breeders to breed high-yield cultivars,it is essential to keep track of the phenotypic data.Unfortunately,phenotyping is usually a manual task that is laborious,expensive and time-consuming[9].More importantly,manually performed visual assessments are subjective,difficult and prone to error [10,11].Therefore,acquiring phenotypes has become a bottleneck in breeding programs [12].There is little doubt that accurate highthroughput phenotyping will accelerate genetic improvements in plants,and shall promote the next green revolution in crop breeding [13].

Previously,phenotypic studies of soybean plants were simple and fragmented,whereas phenomics is based on highdimensional phenotypic data [14].The classification counting and phenotypic measurements of the main stem node,stem,seed and pod have always hindered soybean breeders.For the counting problem,the counted objects are characterized by having a large quantity,variety and high density.For the measurement problem,especially measurements of the projected pod area,manual measurements are almost impossible due to the irregularity of pod’s contours.With regards to phenotype extraction,the most prominent problems are inaccurate counting,heavy counting workload and low measurement accuracy.The most common phenotypic extraction method used by breeders in the early stages of breeding is counting seeds with the naked eye and manually measuring phenotypes with a ruler [15].

Computer vision techniques have greatly improved in recent years,mainly due to the development of neural network techniques such as deep learning [16].Deep learning has shown great potential in terms of image classification,target detection and image segmentation.In particular,instance segmentation has attracted more and more interest in the computer vision community [17].This task has proved to be very challenging,since it requires all objects in an image to be correctly detected while simultaneously segmenting each instance of the object in a precise manner [18].Fast/Faster R-NN [19] and Fully Convolutional Networks (FCN) [20] are used as baseline systems for detection and semantic segmentation respectively.The R-CNN algorithm detects the bounding box of all candidate object regions,and then uses it as a region of interest(ROI)to segment each instance.Most methods that rely on R-CNN use segment proposals to achieve instance segmentation [21-23].

Aich and Stavness [24] used an advanced deep learning architecture to develop a convolutional network for counting leaves.Xu et al.[25] proposed a simple and effective method called Multi-Scale Hybrid Window Panicle Detection,which focuses on enhancing spike features to detect and count a large number of spikelets out in the field.However,the method proposed by Xu et al.[25]is not capable of visually presenting results,and can only use a single function for counting,which is far from meeting the automatic phenotype extraction requirements.An automatic method for pore measurement has been proposed by Song et al.[26],which is based on a mask region-based convolutional neural network(Mask R-CNN) [27] to acquire parameters for stomatal pore anatomy.The Mask R-CNN,based on the Faster R-CNN [28],is an instance segmentation model.Phenotype measurements must segment the pixel area of the target object in an image,and calculate the minimum bounding rectangle for the segmented pixel area.The width and height of the minimum bounding rectangle corresponds to the width and height of the target object.However,when locating the minimum bounding rectangle,the minimumarea rectangle method causes inaccurate positioning [29],which seriously affects the resulting accuracy of measurements.For this reason,we have designed an algorithm which addresses the specific problems that arise from automated phenotype extraction.

We propose an instance segmentation model called Soybean Phenotype Measure-instance Segmentation (SPM-IS) from the integrity and systemic aspects of soybean plants.SPM-IS extracts the phenotype from the entire mature soybean plant.It can measure the projected area of irregular objects that traditional methods find difficult to measure,such as pods and seeds.We then use CNN to automatically identify multiple types of target objects and count and measure them instead of doing so manually.We measured the length of the stems and entire main stems of mature soybeans and counted the main stem nodes and stems,respectively.We classified and counted the mature soybean pods,measured pod width and pod length,and calculated the pod projected area calculation.We then measured the length and width of the seeds,and calculated their projected area and counted them.

2.Materials and methods

2.1.Experimental materials

In this study,20 mature soybean varieties harvested in 2019,including Dongnong 251 (DN251),Dongnong 252 (DN252) and Dongnong 253(DN253),etc.,were selected as experimental materials for phenotypic acquisition,as shown in Table S1.The experimental materials were planted in the Experimental Base of the Northeast Agricultural University of Harbin (45°36'N,126°18'E)in China.The field design was as follows:row length of 2 m,plant spacing of 5 cm,ridge spacing of 60 cm,which was repeated three times.Two hundred harvested soybean plants were used as the experimental materials,and ten strains of each variety were selected as experimental materials for the extraction of the soybean phenotype.In this study,mature soybean plants were divided into their constituent stems,pods and seeds(Fig.1a-e).The stems,pods and seeds were then labeled and input into the neural network for deep learning to take place(Fig.1f).After this,the phenotype of the mature soybean plants was extracted using the trained model (Fig.1g-i).

Fig.1.Phenotype extraction structure diagram.(a) The entire mature soybean plant.(b) Entire disassembled plant,including the complete main stem and pods.(c) Entire main stem after being cut,including the main stem and calibration objects.(d)Pod,including the pod and calibrator.(e) Seeds,including seeds and calibrators.(f)Labeling process of figure data and SPM-IS phenotype extraction.(g)The phenotype of the main stem after extraction.(h)The phenotype of the pod after extraction.(i)The phenotype of the seeds after extraction.

2.2.RGB image acquisition

The RGB image acquisition platform was an LED photo studio,as shown in Fig.S1a,with a size of 80 × 80 × 80 cm.The exterior was made of a black synthetic material,and the interior was made of a silver reflective material.An LED light bar was installed on the top of the photo studio,and reflective materials were provided around the photo studio to ensure that it had sufficient lighting.There was a circular shooting port at the top,and a digital camera was fixed on the circular shooting port (Fig.S1b).Its main parameters are shown in Table S2.In order to prevent the photo from being influenced by background reflections,the background was composed of a light-absorbing cloth.

When choosing a suitable background cloth to ensure an effective image was acquired,a background cloth with a strong color contrast with the subject was selected.In our tests,it was found that the pods and stems were better photographed on a white light-absorbing cloth,and the seeds were better photographed on a black light-absorbing cloth.The entire plant was taken apart before photos were taken,and the pods and main stems were divided into groups.After photographing the pods,the pods were further disassembled into seeds.When photos of the pods were taken,they were randomly scattered across the background cloth.If there was any overlap in the process,they were separated manually.

Datasets were established for the pods,stems and seeds:pod_database,stem_database and seed_database.The images in these three datasets were all in JPG format and taken with a Canon(DS126291) camera at a fixed angle.The direction of the shooting lens and the background cloth was kept vertical to ensure a clear shot could be taken.The size of each image in the pod dataset was 3456×1728,and the image contained a black circular marker and at least five pods (Fig.S2a).The black or white circular object seen in the figure was used as a scale,and was referred to as a marker.A black marker was selected for the white light-absorbing cloth,and a white marker was selected for the black lightabsorbing cloth.The proportional relationship between the marked object and the pixels in the image was used to calculate the actual length represented by each pixel in the figure.The pod_database included 832 photos of pods and markers.The photos contained 832 markers,and a total of 4160 pods.The size of each image in the stem dataset was 1711 × 600,and each image contained a black circular marker and a main stem.The protruding part of the stem was the main stem node,and the part between the two main stem nodes was the stem(Fig.S2b).The stem_database contained 1710 photos,including 1710 markers,4656 main stem nodes and 4556 stems.The size of each image in the soybean seed dataset was 1800×900,and the image contained a white circular marker and several soybean seeds (Fig.S2c).The seed_database contained 665 photos,including 665 markers and 4120 seeds.

The original images in the three datasets were labeled using Labelme software [30],and the outer edges of the target objects were outlined to obtain labeled datasets for neural network learning.For the pod data set,the categories for data annotation were divided into onepod,twopod,threepod,fourpod,and tags.For the stem data set,the categories were divided into stems,stem nodes and tags.Finally,the seed data set was classified as seeds and tags.The distribution of the training set and test set in the data set was 0.7 and 0.3,as shown in Table S3.

Fig.S3 shows an example of pod annotation.For the target object,the green dots were connected to outline the contour of the target object.After completing the outline,a classification name was entered into the dialog box on the right.When all of the target objects were finished,Labelme generated a JSON file as an annotation.The seeds and stems were marked in the same manner,as shown in Figs.S4 and S5.

2.3.Creation of the SPM-IS phenotype extraction network

During the phenotype extraction process,random and complicated scenarios have to be considered,meaning the placement of pods,seeds and stems have different angles(Fig.S6).Additionally,they can appear anywhere in the image in any position and orientation.These complex scenarios produced many challenges for the neural network model’s segmentation accuracy.For complex conditions,an accurate,efficient and stable phenotypic extraction network is absolutely necessary.

YOLACT shows a very robust performance in instance segmentation and an excellent performance in high precision detail [31].SPM-IS is an improvement based on the YOLACT structure that allows it to extract soybean phenotype information (Fig.2).SPMIS includes two parts: feature extraction (Fig.2a) and phenotype extraction (Fig.2b).The feature extraction module (feature backbone) uses a CNN with a Feature Pyramid Network (FPN) [32] to extract the phenotypic features of soybeans.The FPN in this module used both high-resolution low-level features,and highsemantic information of the high-level features.It used these to make a prediction by fusing the features of these different layers.This prediction was made separately on each fused feature layer,which is unlike the conventional feature fusion method.

In the phenotype extraction phase,the mask is obtained by extracting the features.The contour of the target object is outlined on the original image.Then the phenotype is extracted after Principal Component Analysis (PCA),dimensionality reduction and boundary point positioning are carried out on the contour.This process is described in detail in a later section of this report.SPM-IS not only accurately counts the target,but also accurately segments the target object.The actual phenotype data of the object to be measured can be acquired by calculating the proportional relationship between the marker and the object to be measured within the image.This method is also used to extract the phenotype of soybeans,pods and stems.

Lclsis the class confidence loss,where i is the index of the anchor in a mini-batch,and piis the predicted probability of anchor i as an object.The value of pithat represents the ground truth label is 1 if the anchor is positive,and 0 otherwise [33]:

Lregis the box localization loss,is a fourdimensional vector that represents the parameterized coordinates of the predicted bounding box,andis the ground-truth anchor related to the positive anchor.The default value is σ=3:

Lmaskis the binary cross-entropy loss [26],which is defined as:

where s is the true binary mask which arises from manual labeling,and s*is the predicted binary mask.

Fig.2.SPM-IS structure diagram.(a) Feature extraction,including the feature backbone and feature pyramid to extract features.It is then fused with Protonet through the prediction head and non-maximum suppression.(b)The image is processed after the features are extracted,and the mask is extracted after the crop,threshold and contour processes.After a series of operations,such as PCA dimensionality reduction,boundary point positioning and phenotype extraction,the extracted phenotype image is generated.

The focal loss(FL)down-weights the contribution of easy examples and enables the model to focus more on learning difficult examples.In the above y ∈ ｛±1 ｝ specifies the ground-truth class and p ∈（0，1）is the model’s estimated probability for the class with label y=1,γ>0.When γ=1,the focal loss works like the cross-entropy loss function.Similarly,α generally ranges from[0,1].This can be set using the inverse class frequency,or treated as a hyperparameter [34]:

SPM-IS was written in Python,and run in the deep learning framework PyTorch.PyTorch is based on Torch,and was launched by the Facebook Artificial Intelligence Research Institute (FAIR)[35].The details of the software and hardware used in the experiment are provided in Table S4.

2.4.Designing an algorithm to accurately and efficiently locate the minimum bounding rectangle of the target

At present,widely used methods for finding the minimum bounding rectangle of an object include the rotation method and vertex chain code method [36-38].The algorithm minAreaRect provided by OpenCV [29] is the most widely used method,and is more suitable for relatively regular objects.The principle of this approach is first to identify the convex hull of the object which needs to be measured,and then use the rotating calipers method to find the minimum bounding rectangle of the convex hull [39].However,some of the mask’s edge information is lost when the convex hull is calculated using this algorithm.The contours of the pod,stem and seed are nearly smooth,without too many bumps.Meanwhile,the width and height of the circumscribed rectangle with the minimum area are not necessarily the width and height of the object to be measured.Fig.3c shows that minAreaRect does not work with irregular objects,and the correct rectangle for the pods is shown in Fig.3a.The minimum area method is effective when used for completely symmetrical objects;however,the objects measured in the present study are random and irregular.Moreover,the rotation method is not very efficient,and has no targeted search for the minimum bounding rectangle.Therefore,it is not suitable for phenotypic calculations.

To summarize,we propose a soybean phenotype extraction algorithm that can avoid the errors in Fig.3c.Our idea is an algorithm for the mask which uses PCA [40] to reduce the dimensionality of the mask’s contours to obtain the center point and main direction (PCA1 and PCA2) of the object’s contours,and the two orthogonal directions (Fig.3f).Following this,the point with the largest distance from the direction axis along the contour points is searched for,of which there are four.According to these four points,the width and height of the minimum bounding rectangle of the object can be determined (Fig.3g).

After the contour points are reduced via PCA,the center point of the contour and the two orthogonal direction axes can be obtained,which are given by Algorithm 2(Fig.4b).The direction axis is a linear function,and moves up and down in the orthogonal direction of the direction axis.After this movement,the tangent point between the direction axis and the contour is the point at which the maximum distance from the central axis is reached,and also the boundary point of the contour’s minimum bounding rectangular box(Fig.3h).At this point,the contour’s tangent is the direction axis after the movement,which is y=k1x+b*,y=k2x+b*,and b*is the maximum or minimum value:

The contour is the set of boundary points and Ks=｛k1，k2｝is the k value corresponding to the two direction axes.Four boundary points are then used to calculate the width and height of the object.This process is implemented in Algorithm 1(Fig.4a).Algorithm 2 is the main function in Algorithm 1 (Fig.4b).The main process is shown in Fig.3B.

Algorithm 1 is used to calculate the width and height of the minimum bounding rectangle of the object to be measured in the figure with the mask.The process is as follows: find the contour of the mask using FindContour,and calculate all contour points N;obtain the k value of the two direction axes expressed by Ks,and the center points Cx and Cy using GetOrientation;obtain the b value of the linear function using the LineFunction function,the principle if which is given by Eq.(7);calculate the maximum off-axis distance point corresponding to each k value;then obtain the width and height of the minimum bounding rectangle using GetDistance.

Fig.3.Calculating the width and height of the object to be measured.(A)Minimum bounding rectangle.(a)Correct minimum bounding rectangle.(b)Original image of the pod.(c)Incorrect minimum bounding rectangle calculated using minAreaRect.(B)Flow chart showing the extraction of the width and height of the pod.(d)Original image of the pod.(e) Results obtained after extracting the mask from the SPM-IS skeleton.(f) After PCA dimensionality reduction is performed on the mask,the red dimensionality reduction area is obtained.The yellow line represents the height direction and the blue line represents the width direction.(g)Find the point with the largest off-axis distance in the contour.There are four such points.(h,i)Calculate the position,width and height of the minimum bounding rectangle based on the coordinates of the four points.(C)Calculating the width and height of the pod.(j) Original image of the figure pod.(k) Process of measuring the length and width of the pod,where the red lines indicate the length and width,and the green line indicates the minimum bounding rectangle.(l) The image is the result of SPM-IS phenotype extraction.

Algorithm 2 explains the GetDirectionCenter function in Algorithm 1.The first step is to convert the image data into a format that can be operated mathematically.Next,analyze the principal component of the contour to obtain the main direction of the object.Then,obtain the position of the center point and the k value of the direction axis via calculation,and express it as KS before returning the calculated values.

The original image of the pod is shown in Fig.3j,which has a non-horizontal position.Therefore,the calculation method for the width and height of the pod in a non-horizontal scenario is to find the minimum bounding rectangle of the object to be measured,before calculating the width and height according to Algorithm 1 (Fig.3k).The red lines in the figure are the width and height of the pod in the image.The actual calculated length and projected area are shown in the information box in Fig.3l.

2.5.Evaluation indices

To evaluate and predict the results and performance of the ground truth model,functions including precision,recall and degree of integration (IoU) are defined to evaluate the stability of the model [27].Precision and recall are widely used to evaluate the performance of object detection methods.Precision,recall and mAP are defined as follows:

Fig.4.The algorithm can determine the width and height of the minimum bounding rectangle of the object.(a) Width and height of minimum bounding rectangle (b)Algorithm 2 explains the GetDirectionCenter function in Algorithm 1.

where true positive (TP) indicates that the objects are correctly identified by the model from the defined object region;false positive (FP) indicates that the background is misidentified as objects;and false negative (FN) indicates that objects are misidentified as the background.

The IoU index can be used to compare similarities and differences between finite sample sets.The greater the IoU coefficient,the higher the similarity between the segmentation result and the corresponding ground truth.

The IoU index is defined as follows:

where Spredis the detection result location and Sgtis the real object location.

The evaluation index used in this study is the standard metrics of the MS COCO [41].For the primary AP,0.5:0.05:0.95 means starting from IoU=0.5,with increments of 0.05,until an IoU=0.95 is reached.These result in computations of AP thresholds at ten different IoUs.

3.Results

3.1.Choosing a suitable CNN model

In order to find the best CNN algorithm for the segmentation of different instances of soybeans,pre-training of four CNN models was carried out.For 60,000 iterations,the experimental data of the mask mAP,box mAP and frames per second (FPS) of the four models were calculated,as shown in Table S5.The performances of the four models were then subsequently compared.FPS was the fastest in the case of the Resnet-101-FPN model with a processing size of YOLACT-400,which reached 45.3 FPS.

In terms of mAP,Resnet-101 showed the best performance among the three models because it had more parameters than the other two.In the case of the Resnet-101-FPN model and YOLACT-700 processing size,the mAP of the box was 95.4,and the mAP of the mask reached 95.8.The higher the mAP,the slower the processing speed.In the case of the Resnet-101-FPN model and YOLACT-700 processing size,only 23.4 FPS was reached,which was half that of YOLACT-400.Considering the phenotype extraction was more focused on accuracy mAP,the Resnet-101-FPN model and YOLACT-700 size were chosen.The detailed hyper-parameter list of the model is shown in Table S6.

3.2.Model training and evaluation

The mAP and loss graphs for detecting pods,seeds and stems were drawn to evaluate the performance of the model during the training process.For the box localization loss,class confidence loss,mask loss and focal loss,the loss decreased rapidly during the initial 10,000 iterations.Following the initial 10,000 iterations,the loss function fluctuated less and less,and the loss function eventually converged to 0 (Fig.5).During the training process of the pod recognition model,the mAP of the box gradually improved as the number of iterations increased.

There was an obvious shock at 3000 iterations (Fig.5h),which may have been related to the selection of training samples.In this batch,the characteristics of randomly selected samples were similar,causing local overfitting to occur.In the following multiple training,a diverse range of sample types were selected so that the local overfitting situation did not occur.The amplitude of the mask loss and class confidence loss of the stem greatly fluctuated relative to the pod during the training process(Fig.5j and k).For the stem,the feature of the boundary between the stem and the stem node was not markedly apparent.This led to large fluctuations in mask loss during the training process.The texture characteristics of the stems and stem nodes were similar,which increased the difficulty of the category judgment.

During the training process of the seed recognition model,as the iteration times increased,the mAP of the box gradually improved.However,the rising speed was relatively slow compared to the pod recognition model.The mAP of the mask improved the most between 0 and 10,000 iterations.Additionally,the ascent speed slowed down and then subsequently increased (Fig.5n).

Fig.5.The training process of the pods,stems and seeds model.(a,b) The mAP change diagram of the box and mask of the pod during the training process.(c-f) The loss change diagram of the pod during training.(g,h)The mAP change diagram of the box and mask of the stem during the training process.(i-l)The loss change diagram of the stem during training.(m,n)The mAP change diagram of the box and mask of the seed during the training process.(o-r)The loss change diagram of the seed during training.

The amplitude change in the mask loss was larger than that for the pod,and the changing pattern was similar to what was observed in the stem training(Fig.5p).The proportions of an individual pod,seed and stem in the image were compared,and a single seed was shown to have the smallest proportion in the image.Perhaps because small target objects have fewer features,the CNN model takes longer to learn about small target objects.The judgment of the category in the seed is simple,and only involves the background and seed judgment.Therefore,there were no large fluctuations in the training process for the seed.

We then tested the trained models on the pod test set,stem test set and seed test set.The test results showed that the box mAP of the pod,stem and seed were 95.3%,93.7% and 94.9% respectively.The test results showed that the mask mAP of the pod,stem and seed were 95.7%,93.5% and 94.6% respectively.Table S7 shows the mAP of the pods,seeds and stems.

3.3.Phenotype recognition results

The segmentation of soybean pods,stems and seeds through the network designed in this study (SPM-IS) was performed very well,and the segmentation results were displayed visually.

The recognition results for the pod are shown in Fig.6a.The upper left corner shows the respective count values of onepod,twopod,threepod and fourpod in the image.To recognize different pods,masks of different colors were used for covering and segmentation.The first item of the recognition information is the classification,and the probability of the classification is between 0 and 1.PA is the projected area value of the mask for pod detection.W represents the width of the pod,and H represents the height of the pod.The round object is a marker.

The results of the stem recognition are shown in Fig.6b,whereby the upper left corner of the image is the count value of the stems and nodes.The recognition of the main stem node and the stem is divided according to the different colors of masks.The first item of the recognition information is the classification and probability of classification,which is between 0 and 1.Fig.6b shows that,although the posture of the stem is curved,SPM-IS divided the stem very clearly.

The SPM-IS recognition results for the seeds are shown in(Fig.6c),whereby the upper left corner shows the total number of seeds.The recognition of different seeds was segmented according to different colors of masks.The first item of the recognition information is the classification,and the classification value is between 0 and 1.PA is the projected area of the mask used for seed detection,and W and H are the width and height of the seed respectively.

3.4.Evaluation of SPM-IS measurement performance

Correlation coefficient analysis was performed on the phenotype data extracted by SPM-IS,and the data was manually measured to evaluate the reliability and stability of SPM-IS for extracting the pod phenotype.The traits were manually obtained with vernier calipers and tape measures (Fig.S7).A total of 130 pods were selected,and the pod length and width data obtained via manual and SPM-IS measurements were correlated.Following this,a scatter plot was drawn.The R2of the pod length was 0.9755 (Fig.7a);the R2of the pod width was 0.9872 (Fig.7b).

A total of 130 stems and 100 complete main stems were selected.The correlation between the stem length and complete main stem length data obtained via manual and SPM-IS measurements were analyzed,and a scatter plot was drawn.The length of the complete main stem was measured by adding up the lengths of all the stems and the main stem nodes in turn.The R2of the stem length was 0.9692 (Fig.7d);the R2of the complete main stem length was 0.9803 (Fig.7e).

The correlation between the manual measurement and SPM-IS measurement data for the length and width of 130 seeds was analyzed,and a scatter plot was drawn.The scatter plot of the seed length is shown in Fig.7g,whereby R2was 0.9656.The scatter plot of the seed width is shown in Fig.7h,whereby R2was 0.9716.The correlation was very strong,and could be used to replace manual measurements.The correlation coefficient of the seed was lower than that of the pod,which may be due to the pod being larger than the seed.

The sizes of the yellow dots in Fig.7c,f and i are related to the number of occurrences of this point.The more occurrences there were,the larger the point is.The correlation between manual counting and SPM-IS counting was then analyzed.The correlation coefficients of the pods,stems and seeds were 0.9733,0.9872 and 0.9851 respectively (Fig.7c,f and i).It can be seen that the points on the straight line are huge,which also denotes a higher accuracy.The deviating points of the pods are smaller than those of the seeds because the number of pods in the figure is small(Fig.7c).The distribution in the number of seeds is relatively uniform,and the outliers are small (Fig.7i).

4.Discussion

4.1.SPM-IS: A good replacement for humans

The results of the segmentation measurements were analyzed and found to be rather satisfactory.The correlation coefficients were all greater than 0.96,as shown in Fig.7.The correlation was strong,and the results showed that SPM-IS could replace manual measurements of stems.Table S8 records the time taken for the manual and SPM-IS methods to obtain the phenotype when acquiring phenotypic data,including the image acquisition process,model training and test process,and trait extraction process.Through comparing the two methods,we found that SPM-IS can save a significant amount of time.In terms of measurement performance,the measurement of phenotype was proven to be more effective than that achieved by Song et al.[26]

4.2.Effect of shadows on segmentation results when collecting photo data

The observation and analysis of the results of the SPM-IS phenotype experiment demonstrated that the most crucial factor that affected the mAP in the experiment was the shadow produced by light when taking photos.The observation and analysis of the phenotypic results demonstrated that the SPM-IS method was not very good at distinguishing shadows.This was especially the case when distinguishing between stems and shadows.In Fig.8b,the true dividing line between the stem and the shadow is denoted by the yellow line.Because the color of the stem and shadow are rather similar,SPM-IS could not distinguish between the two.When they overlapped,this impacted the results of segmentation.

Comparing the influence of shadows on the pods (Fig.8a),the results show that SPM-IS could distinguish between shadows,but not entirely.This might be related to the color of pods.In Fig.6b,SPM-IS was able to distinguish between shadows,which may be related to the contrast between the color of the target object and the background.The results also showed that SPM-IS had an unexpected segmentation performance during seed segmentation (Fig.8c),which might be related to photographing the seeds with a black background cloth.This also confirms the theory that contrast has an effect on segmentation.

Fig.6.Phenotypic data extracted by SPM-IS.(a) The number of onepods is one,which is displayed in green.The number of twopods is two,which are displayed in red and blue.The number of threepods is one,which is displayed in blue.The number of fourpods is one,which is displayed in yellow.(b) There are three stems,displayed in blue,green and blue,and the two main stem nodes are displayed in yellow and orange.(c)Phenotypic recognition results of the seeds show that the total number of seeds is 7,and the superscript of each seed displays the phenotypic information of the seed.

Fig.7.Correlation analysis of SPM-IS measurement and counting.(a-b) Correlation analysis between SPM-IS and manual measurement of pod length and pod width.(c)Correlation analysis between SPM-IS and manual counting of the number of pods.(d) Correlation analysis between SPM-IS and manual stem length measurement.(e)Correlation analysis between SPM-IS and manual measurement of entire stem length.(f)Correlation analysis between SPM-IS and manual counting of the number of stems.(g-h)Correlation analysis between SPM-IS and manual measurement of seed length and width.(i)Correlation analysis between SPM-IS and manual counting of the number of seeds.

An important question that must be asked is how can errors introduced by experimental equipment be effectively avoided when taking photos.This question is worthy of further study.In the present study,the shadow and texture of the stems were similar,which caused misjudgments to be made.According to related studies,CNN training actually learns texture features(texture bias)rather than shape features [42].This is different from human cognitive methods,where there is a texture bias which causes mask outline errors.Therefore,to improve the accuracy of discrimination,shape bias must be highlighted.Visualization of Stylized-ImageNet (SIN) was created by applying an adaptive instance normalization(AdaIN) style transfer to ImageNet images.SIN training with mixed raw data can simultaneously extract both texture and shape features,which conforms to human intuition and improves the accuracy and robustness of various tasks [42].Yamane et al.[43] segmented photos of concrete structures under adverse conditions,such as shadows and dirt,achieving excellent results which inspired us to improve the segmentation performance by improving the network.

4.3.The SPM-IS soybean phenotype extraction method is beneficial to crop breeding

High accuracy.The manual extraction of phenotypes is inevitably affected by people’s subjective ability.This artificially introduces some inestimable errors,which are then brought into the experimental data.This affects the final conclusion when analyzing the data.Fortunately,these errors can be effectively avoided by employing some measure.Because SPM-IS is objective,it can perform its tasks accurately according to artificial settings.

Labor-saving.SPM-IS can be used to extract soybean phenotypes with a high throughput.The entire process can be automated,including the generation and collation of phenotypic data.SPM-IS is also a method that frontline breeding experts have hoped for,and can be used to replace the manual method [13].

Fig.8.Influence of shadows on SPM-IS phenotype extraction.(a) Pod shadows.(b) Stem shadows.(c) SPM-IS segmentation of seeds.

Our experimental group,the Plant phenomics and Genomics Lab (PPGL),have already started developing the web version of the phenotype extraction platform.Users can simply upload image data to obtain the corresponding phenotype data.This platform can be utilized from fields and laboratories with an active internet connection,using a smartphone or computer.The core algorithm embedded in the online phenotype extraction platform is the result of the research in this paper.Its primary purpose is to apply the algorithm we studied to actual real-world applications.After the test is conducted and deemed to be stable,it will become available for public use and provided to breeders.The ultimate goal of our research is to use advanced technology to provide technical guidance for agricultural production.

CRediT authorship contribution statement

Shuai Li:Formal analysis,Investigation,Methodology,Visualization,Writing -original draft.Zhuangzhuang Yan:Supervision,Validation.Yixin Guo:Investigation,Data curation.Xiaoyan Su:Data curation,Validation.Yangyang Cao:Investigation,Data curation.Bofeng Jiang:Investigation.Fei Yang:Data curation.Zhanguo Zhang:Project administration,Resources.Dawei Xin:Writing review &editing,Funding acquisition.Qingshan Chen:Writing review &editing,Funding acquisition,Resources.Rongsheng Zhu:Conceptualization,Data curation,Funding acquisition,Resources,Writing review &editing.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (31400074,31471516,31271747,and 30971809),the Natural Science Foundation of Heilongjiang Province of China(ZD201213),and the Heilongjiang Postdoctoral Science Foundation(LBH-Q18025).We are grateful to Zhaoming Qi,Chunyan Liu and Xiaoxia Wu for providing help with the materials and methods for this research,and we thank Zhengbang Hu,Yang Li,and Zhenqing Zhao for providing technical and instrumental assistance for this research.

Appendix A.Supplementary data

Supplementary data for this article can be found online at https://doi.org/10.1016/j.cj.2021.05.014.