Liu-Liu Cao, Mei Peng, Xiang Xie, Gong-Quan Chen, Shu-Yan Huang, Jia-Yu Wang, Fan Jiang, Xin-Wu Cui,Christoph F Dietrich

Abstract Artificial intelligence (AI) is playing an increasingly important role in medicine,especially in the field of medical imaging. It can be used to diagnose diseases and predict certain statuses and possible events that may happen. Recently, more and more studies have confirmed the value of AI based on ultrasound in the evaluation of diffuse liver diseases and focal liver lesions. It can assess the severity of liver fibrosis and nonalcoholic fatty liver, differentially diagnose benign and malignant liver lesions, distinguish primary from secondary liver cancers, predict the curative effect of liver cancer treatment and recurrence after treatment, and predict microvascular invasion in hepatocellular carcinoma. The findings from these studies have great clinical application potential in the near future. The purpose of this review is to comprehensively introduce the current status and future perspectives of AI in liver ultrasound.

Key Words: Machine learning; Deep learning; Radiomics; Diffuse liver diseases; Focal liver diseases; Ultrasound

INTRODUCTION

In the past several years, liver diseases have affected millions of lives and became one of the main causes of illness and death in the world[1]. It is reported that more than one-fifth of the Chinese population are affected by liver diseases, such as liver fibrosis, liver cancer and nonalcoholic fatty liver disease (NAFLD), contributing unambiguously to health loss. Therefore, paying more attention to liver diseases is of great significance.

Artificial intelligence (AI) is defined as the research of algorithms that enable machines to have the ability of reasoning and performing functions such as solving problems, recognizing object and word,inferring world states, and making decisions[2]. AI is a precise prediction technique that automates learning and recognizes patterns in data. Apart from this, AI has been extensively applied to medical diagnosis, especially in medical image analysis. This application mainly relies on deep learning, a subfield of machine learning. Deep learning is on the frontier of AI, which is based on deep neural networks (DNNs) with more than one hidden layer. Convolutional neural networks (CNNs) are a branch of DNNs that are particularly useful for recognizing images and have stimulated a large amount of interest from industry, academia and clinicians[3].

Compared to other medical imaging techniques, ultrasound is noninvasive, portable and can provide real-time imaging. In recent years, AI-powered ultrasound has become more developed and been implemented in clinical applications in order to reduce the subjectivity and improve the efficiency of ultrasound diagnosis[4]. Many studies have confirmed the value of AI in the evaluation of thyroid nodule, breast lesion and liver lesion classification by ultrasound. In addition to these applications,other AI applications in ultrasound have also been explored and achieved great progress.

In liver medical imaging, AI can make a quantitative assessment by recognizing imaging information automatically to aid physicians in making more precise and comprehensive imaging diagnoses[5]. This technique has been extensively applied to computed tomography (CT), positron emission tomography-CT, magnetic resonance imaging and ultrasound to diagnose liver lesions. For instance, deep learning based on CT and positron emission tomography-CT can be used to detect new liver tumors and metastatic liver malignancy, and to predict the primary origin of liver metastasis[6-8].

There are also many studies that have illustrated the application of AI in liver ultrasound, while a comprehensive review of AI in this field is lacking. In this review, we will introduce the application of AI based on ultrasound in diffuse liver diseases, including liver fibrosis and steatosis, and focal liver lesions (FLLs), including their differential diagnosis, prediction of a curative effect and prognosis and microvascular invasion (MVI) of hepatocellular carcinoma (HCC). The main structure of this review was illustrated in Figure 1.

APPLICATION OF AI IN DIFFUSE LIVER DISEASE

There are a variety of diffuse liver diseases that can be asymptomatic or cause severe liver dysfunction,and many of them may lead to cirrhosis, hepatic carcinoma, and death. We will introduce the applications of AI based on ultrasound in two common diffuse liver diseases,i.e.liver fibrosis and steatosis.

Liver fibrosis

Liver fibrosis is the early step of cirrhosis and an important pathological basis of HCC[9]. Therefore, the early detection and prevention of liver fibrosis is essential in the clinical setting. However, although liver biopsy is the gold standard for classifying liver fibrosis using the Metavir score[10] or New Inuyama classification[11] to distribute the score ranging from F0 (no fibrosis) to F4 (cirrhosis). The use of tissue examination for the assessment of liver fibrosis is controversial because liver biopsy is invasive and liver fibrosis is not equably distributed throughout the liver. There are an increasing number of credible, noninvasive and available approaches being widely applied in clinical practice. Recently, a large number of noninvasive techniques have been used to prevent adverse outcomes through the application of AI based on ultrasound.

AI based on B-mode ultrasound:As early as 20 years ago, AI was used to assist the diagnosis of liver fibrosis. Badawiet al[12] creatively proposed an approach that employed fuzzy reasoning techniques to identify diffuse liver diseases automatically by using digital quantitative features measured from the ultrasound images. They extracted parameters only from B-mode images, and the results revealed that this approach had higher specificity and sensitivity for the diagnosis of liver fibrosis than the statistical classification techniques, which had a certain effect but could not help much.

Apart from this, a novel deep multi-scale texture network based entirely on B-mode ultrasound images that was proposed recently seems to be more convenient[13]. The area under the receiver operating characteristic curve of this approach was 0.92 for significant fibrosis (≥ F2) and 0.89 for cirrhosis (F4) in the validation group, which outperformed the ultrasonographers and three serum biomarkers during diagnosis. Although it cannot be used for liver fibrosis staging now, it has excellent potential in the future workflow.

AI based on Doppler ultrasound:On the basis of grey-scale parameters from B-mode images, Doppler parameters of intrahepatic blood vasculature were added as essential parameters. Eventually, five ultrasonographic variables, including the liver parenchyma, thickness of the spleen, the hepatic vein waveform, hepatic artery pulsatile index and damping index, were selected as the input neurons. A data optimization procedure was used in an artificial neural network for the diagnosis of liver fibrosis, which achieved an area under the curve of 0.92[14]. Although this model proved to predict liver cirrhosis accurately, it still could not provide a specific grading.

AI based on elastography:In recent years, with the development of ultrasound, studies have proposed computer-aided techniques based on elastography that is of great importance in ultrasound images to identify and stage liver fibrosis. Real-time tissue elastography (RTE) is one of the recently developed elastography techniques. In a study, 11 image features were extracted directly from the RTE software that was installed in the ultrasound system to quantify the patterns of the RTE images[15]. Then, the data were processed and divided among four classical classifiers. The results showed that the performance of the adopted classifiers was much better than the previous liver fibrosis index method,which predicted the stage of fibrosis using RTE images and multiple regression analyses. The good performance in this study demonstrated that machine learning had the potential to perform as powerful tools for staging liver fibrosis.

Nowadays, most applications of AI in evaluating the stage of liver fibrosis are based on shear wave elastography. An automated approach including the image quality check, region of interest (ROI)selection and CNN classification based on shear wave elastography showed a more accurate detection of ≥ F2 fibrosis levels than a previously published baseline approach, with an area under curve of 0.89vs0.74[16]. The deep learning radiomics also presented the potential diagnostic performance in chronic hepatitis B patients compared with two-dimensional shear wave elastography[17]. AI could help stage liver fibrosis more accurately with the assistance of elastography.

Liver steatosis

Hepatic steatosis, characterized by the accumulation of fat droplets in hepatocytes, can develop into nonalcoholic fibrosis, steatohepatitis, cirrhosis, and even HCC[18,19]. Early detection and treatment may halt or reverse NAFLD progression[19]. As a consequence, there is a critical need to develop noninvasive imaging methods to assess hepatic steatosis. Noninvasive liver imaging methods including CT, magnetic resonance imaging and ultrasound have been extensively investigated[20].

Ultrasound is the first-line examination for identifying liver steatosis. It shows an enlarged liver with a greater number of echoes caused by fat droplets, and the liver is brighter and more hyperechoic compared with the right kidney. The image is qualitative and relies on the subjective judgement of the operator, which definitely leads to variable results and low reproducibility[21]. To overcome the observer bias, a series of quantitative and semi-quantitative parameters, including attenuation and backscatter coefficients, the hepato-renal index (HRI) and ultrasound envelope statistic parametric imaging (known as speckle statistics), have been implemented, some of which represent excellent reproducibility and reliability[21-24]. At present, almost all the studies published have concentrated on NAFLD.

It was reported that the detection of moderate and severe steatosis based on ultrasound has an 84.8%sensitivity and a 93.6% specificity, while mild steatosis has an even lower sensitivity[25]. Recently, some researchers applied AI to improve the ultrasound detection rate of NAFLD, and the results were promising. Table 1 shows the studies using AI based on ultrasound to identify steatosis. The studies showed that AI had tremendous potential in helping diagnose liver steatosis. Some studies attempted to optimize CNN models. In the future, classifying the degree of liver steatosis with the assistance of the AI could be a trend.

Qualitative evaluation:Deep learning has been applied to qualitatively evaluate NAFLD. An approach of assessing fatty liver disease by utilizing deep learning based on CNNs with B-mode images was proposed[26]. The study recruited 135 participants with known or suspected NAFLD to investigate the function of four liver views (three views in the transverse plane, including hepatic veins at the confluence with the inferior vena cava, right portal vein and right posterior portal vein, and one view in the sagittal plane,i.e.the liver and kidney view) in the assessment[27]. The study assessed attention maps for liver assessment based on CNNs, which illustrated that the available image features provided by each view could assess liver fat. Unlike the previous study, magnetic resonance imaging proton density fat fraction was used as a reference standard, which was not precise enough compared with liver biopsy.

A novel framework combining transfer learning with fine-tuning was proposed[28]. Although this study revealed that the new framework outperformed CNN, this conclusion was not entirely convincing because the radiologists’ qualitative scores were the reference standard. This framework was also utilized in other studies and achieved a good performance.

With the development of deep learning, Chouet al[29] established two-class, three-class and fourclass prediction models to classify the severity of steatosis by using B-mode ultrasound images from 2070 patients. Although liver biopsy is the gold standard, the deep learning model could select eligible patients for a liver biopsy by evaluating the severity of fatty liver preliminarily, which would reduce unnecessary tests.

Different deep learning algorithms tend to have different performances. The combined deep learning algorithm based on B-mode images had an area under the receiver operating characteristic curve of 0.9999 and accuracy of 0.9864 compared with other algorithms[30]. Therefore, in future studies selecting an optimal algorithm is important.

Quantitative evaluation:AI has also been applied to quantitatively evaluate NAFLD. Radiofrequency signals may negate the loss or change of data during translation to B-mode ultrasound images. A study acquired the diagnosis and fat fraction of NAFLD by inputting the original data based on onedimensional algorithms[31]. The investigators obtained a 97% sensitivity and 94% specificity, and the positive predictive value was up to 97%, which demonstrated that the utilization of original ultrasound radiofrequency could be applied in diagnosing NAFLD and to quantifying liver fat fraction. Similarly,in an animal experiment, a CNN model based on radiofrequency signals was shown to have a betterperformance than the traditional quantitative ultrasound when classifying steatosis[32].

Table 1 Studies using artificial intelligence based on ultrasound for fatty liver disease diagnosis

An HRI model based on CNNs was also tested for NAFLD evaluation. Chaet al[33] reported that an automated approach had no significant difference in hepatic measurements and HRI calculations compared with experienced radiologists, which indicated that the aid of deep learning could reduce a radiologist’s workload and improve the residents’ diagnostic accuracy. In this study, an automated HRI calculation algorithm was used, including liver and kidney segmentation, kidney ROI extraction, liver ROI extraction, and calculation of the HRI.

APPLICATION OF AI IN FLLS

HCC is the most conventional original malignant FLL, which is the sixth most common cancer in human beings as well as the fourth cause of cancer-related deaths in the world[1]. Hence, early accurate differential diagnosis of malignant and benign FLL is important for the management and prognosis of patients[34].

Ultrasound is the first-line imaging modality to identify FLLs in the clinical workflow. The development of AI provides a new method to improve the accuracy of ultrasound in diagnosing FLLs.Compared with radiologists viewing anatomical images, AI can better reflect monolithic tumor morphology as well as capture both granular and radiological patterns in a specific task, which is difficult by normal human vision[35]. Figure 2 illustrates the flowchart of the application of deep learning and radiomics in FLLs. Studies have confirmed that the application of AI can improve the diagnostic performance of ultrasound for FLLs (Table 2).

Differential diagnosis of FLLs

AI based on B-mode ultrasound:AI has been widely used in differentiating malignant and benign FLLs based on B-mode ultrasound. Gray level co-occurrence matrix could be used in extracting features from B-mode images, which has been used in differentiating malignant and benign FLLs combined with a fuzzy support vector machine[36]. This study achieved an area under the curve of 0.984 and 0.971 in database 1 and database 2, respectively, which confirmed the feasibility of AI in this field.

With the development of AI, deep learning plays a more vital role in the differential diagnosis of FLLs. A CNN with ResNet50 was utilized to recognize benign from malignant solid liver lesions through ultrasonography. The performance was comparable to that of expert radiologists[37]. However,this study did not involve other information such as clinical factors. In another (multicenter) study, after adding seven clinical factors, a higher accuracy, sensitivity and specificity was obtained, compared with those from radiologists with 15 years of experience. The area under the curve for recognizing malignant from benign lesions reached up to 0.924 in the external validation cohort[38].

Figure 2 Illustration of the flowchart of the application of deep learning and radiomics in focal liver lesions. These two methods were based on big data, which contained image preprocessing, feature extraction and model construction.

However, the aforementioned studies only included common FLLs. Increasing the types of FLLs may confuse the diagnosis and reduce the accuracy. Similar to the previous study, a multicenter study estimating internal validation and external validation cohorts had a larger volume of training data and involved more varieties of FLLs, including cysts, HCC, hemangiomas, focal fatty infiltration and focal fatty sparing[39]. Although they obtained a lower sensitivity due to more types of diseases, the performance in the external validation cohorts was still satisfactory. In addition, they utilized videos as training materials to achieve real-time analysis in the future workflow. This novel approach would offer great convenience to radiologists in helping differentiate FLLs.

AI could also be used in the classification of FLLs. In order to optimize feature sets, a hybrid textural feature extraction system was proposed by Hwanget al[40]. In their preliminary study, a high accuracy was observed in classifying cystsvshemangiomas and cystsvsmalignant lesions. However, when classifying hemangiomasvsmalignant lesions by extracting multiple ROI, the accuracy was only 80%.The proposed approach exhibited a better accuracy in all classification groups by quantifying the key features in ultrasound images, especially in classifying hemangiomavsmalignant, with an accuracy of 96.13%.

Later, a sparse autoencoder system based on deep learning was proposed in diagnosing cysts,hemangiomas and malignant lesions, and it outperformed the three progressive techniques, including K-enarest neighbor, multi-support vector machine and naive Bayes, with an overall accuracy of 97.2%[41].

These two studies[40,41] focused on three kinds of FLLs. An algorithm that could simultaneously detect and characterize FLLs based on deep learning was proposed in diagnosing HCC, focal nodular hyperplasia, cysts, hemangiomas and metastasis, and it achieved an average area under the curve of 0.916[42]. That study yielded promising results by using a small amount of data. Larger databases would increase the accuracy of this model.

AI based on contrast-enhanced ultrasound:It is reported that contrast-enhanced ultrasound (CEUS)images had better sensitivity and specificity for differentiating between malignant and benign tumors compared with B-mode images, which indicated CEUS had a superior diagnostic performance.Combining AI with CEUS could differentiate benign and malignant FLLs and classify different kinds of malignant lesions.

AI could be used to differentiate malignant and benign FLLs based on three-phase CEUS images. A two-stage multiple view learning that represented the integration of deep canonical correlation analysis and multiple kernel learning was used to fuse the characteristics of three-phase patterns in CEUS,presenting an accuracy of 90.41%[43]. The proposed algorithm had both a low computational complex and a high predictive accuracy. For multiview CEUS images, utilizing a multimodal feature fusion algorithm is necessary.

Compared with deep canonical correlation analysis-multiple kernel learning, the use of a threedimensioned CNN, which integrated the relationship between two temporally adjacent frames to extract features spatially and temporally, achieved a higher accuracy of 93.1%, sensitivity of 94.5% and specificity of 93.6%[44]. However, this algorithm still needs to be validated.

These two studies above[43,44] exploited heterogeneous visual morphology to describe the difference between liver masses. Apart from this method, time-intensity curve (TIC), which represents the contrast intensity constantly and generates the fitted curve of enhanced intensity during the process, was used in many studies.

Support vector machine[45,46] and deep learning[47,48] based on TICs have presented good performances in differentiating FLLs. A support vector machine-based image analysis system was used for FLL classification and presented an area under the curve of 0.89[45]. An artificial neural network diagnostic system based on TICs was shown to have a similar accuracy and specificity in classifying five different liver tumors 10 years ago[47]. Later, deep learning became more mature, and TICs of the arterial and the portal vein phases of CEUS videos were extracted on the basis of the deep belief networks, a kind of neural network that was composed of layers of Boltzmann machines, to analyze the extracted TICs[48]. The accuracy of this deep learning method for classifying benign from malignant lesions was 83.36%. A novel evaluation procedure named ‘leave-one-patient-out’ and custom DNNs were creatively presented. That study involved various types of liver lesions and compared the custom DNN designs with the state-of-the-art architectures and obtained a maximal accuracy of 88% by utilizing the proposed evaluation procedure in both pretrained and trained-from-scratch models. This novel approach has a magnificent prospect for development, and it is worth further investigation.

AI based on CEUS was shown to assist in clinical settings as the reference and improve the performance of residents in the differentiation of benign and malignant FLLs[49]. In the future, it will likely play a supporting role in clinical work. However, AI based on TICs tends to complicate the calculation because generating TICs is a time-consuming process. Therefore, developing new approaches to extract features from CEUS images is important.

Application of AI in predicting curative effect and prognosis of HCC

It was reported that the Edmondson-Steiner grade is a vital preoperation predictor of tumor survival and recurrence after undergoing surgical resection[50,51]. Because preoperative pathological differentiation grade can only be obtained by an invasive biopsy[52], it is necessary to explore a noninvasive method to predict therapeutic effect, recurrence and metastasis to achieve personalized treatment.

Some studies[53-55] demonstrated the superiority of AI based on CEUS in predicting a curative effect and prognosis in HCC. Although the results revealed a better performance of AI models compared with a single clinical or ultrasound model, a better performance may be obtained by adding clinical factors in the future studies.

Predicting curative effect of HCC:Transarterial chemoembolization (TACE) is the first-line therapy in patients who are diagnosed with mid-stage HCC, and the response to the first TACE treatment is related to the subsequent curative effect and survival. Therefore, it is necessary to predict the personalized responses to the first TACE treatment. A deep learning radiomics-based CEUS model, machine learning radiomics-based B-mode image model and machine learning radiomics-based TIC of the CEUS model were established to achieve this function[53]. These models presented a better performance compared with the hepatoma arterial embolization prognostic score based on three indexes concerning liver function and tumor load, which will be of great benefit in selecting both first treatment and subsequent therapies after the first TACE treatment.

Predicting prognosis of patients with HCC:Radiofrequency ablation and surgical resection are recommended for early-stage HCC. Deep learning could also be used to predict the progression-free survival of these two therapies in HCC patients[54]. Two models based on these two kinds of therapies provided a satisfactory prediction accuracy and calibration of 2-year progression-free survival. In another study[55], a three dimensional-CNN model, which avoided missing information from CEUS images compared with extracting features from four-phase images, was used. It was observed that predicting prognosis of different treatments in advance and timely swapping of treatment would increase the 2-year progression-free survival, which could contribute to a better prognosis.

Application of AI in predicting MVI of HCC

MVI has shown to be the independent predictor of recurrence and poor outcomes of HCC. Therefore,making a noninvasive and accurate preoperative identification of MVI would be of great significance for HCC patients. The application of AI in predicting MVI achieved good performance based on gray-scale ultrasound images and CEUS.

The radiomics score based on ultrasound of HCC was established and shown to be an independent predictor of MVI[56]. The performance of the clinical nomogram improved significantly with the aid of a radiomics score, which demonstrated the important role of this technique.

Features of the peritumoral area have been shown to be more accurate recently[56]. The radiomic signatures from the gross and peritumoral region showed the best performance compared with the gross-tumoral region and the peritumoral region[57]. The area under the curves were 0.726 based on features of the gross and peritumoral region and 0.744 after the incorporation of essential clinical information.

These studies[56,57] mentioned above have confirmed the application of AI based on gray-scale ultrasound, and AI could be applied to CEUS in predicting MVI as well. Zhanget al[58] extracted radiomics features from the B-mode, artery phase, portal venous phase and delay phase images of preoperative CEUS to construct four radiomics scores based on the primary dataset. Then, they used four radiomic scores and clinical factors for multivariate logistic regression analysis, which demonstrated that the portal venous phase and delay phase radiomics score, tumor size and alphafetoprotein level were independent risk factors in predicting MVI. The radiomics nomogram based on these four predictors indicated a better discrimination and a good calibration compared with the clinical model (based on tumor size and alpha-fetoprotein level) in both the primary dataset (area under the curve: 0.849vs0.690) and the validation dataset (area under the curve: 0.788vs0.661). This study developed a new noninvasive predictive nomogram based on CEUS that could provide useful information in predicting MVI preoperatively, thus facilitating the choice of a more appropriate surgical option.

CONCLUSION

In conclusion, AI can provide great assistance in the evaluation of diffuse liver diseases (including liver fibrosis and liver steatosis) and FLLs. First, it could be applied to identify and stage liver fibrosis on the basis of B-mode ultrasound, Doppler ultrasound, and elastography. Second, the application of deep learning could be used to make qualitative evaluation based entirely on B-mode images and quantitative evaluation based on radiofrequency signals and HRI, which would improve the ultrasound detection rate of NAFLD. Third, AI has the ability to differentiate malignant FLLs from benign FLLs as well as classify different kinds of FLLs with a better performance compared with clinical indexes.Fourth, the curative effect and prognosis of HCC treatment can be predicted, and an optimal personalized treatment can be chosen. Last, AI based on B-mode ultrasound and CEUS could predict MVI of HCC preoperatively, which could be helpful for more appropriate surgical planning. These applications had good specificity, accuracy and a comparable or even better performance compared with experts in the diagnosis and differentiation of diffuse and focal liver lesions.

There are also some limitations in the applications of AI using ultrasound. First, it is difficult to prepare a large-scale dataset, especially for medical images. Second, although deep learning is the widest used algorithm and has good performance in various studies, its interpretability and generalization is low. Third, the input data may vary from different equipment and operators, which would influence the performance of AI. Last, because a large amount of data is needed to train and validate the established algorithms, the conclusions of many single-center studies are not convincing. Therefore,researchers are expected to conduct more multicenter studies and incorporate more samples, as much as possible. At the same time, optimizing algorithms and creating standards for medical images are also necessary. Despite medical images, researchers could also build the database containing important clinical factors to establish a more comprehensive AI model for future work.

FOOTNOTES

Author contributions:Jiang F, Cui XW and Dietrich CF conceived and established the design of the paper; Cao LL,Peng M, Xie X, Chen GQ, Huang SY, Wang JY, Dietrich CF, Jiang F and Cui XW explored the literature data; Cao LL provided the first draft of the manuscript, which was discussed and revised critically for intellectual content by Peng M, Xie X, Dietrich CF, Jiang F and Cui XW; all authors discussed the statements and conclusions, and approved the final version to be published.

Supported byNational Natural Science Foundation of China, No. 82071953; and 2020 Hunan Province Clinical Medical Technology Innovation Guidance Project.

Conflict-of-interest statement:All the authors report no relevant conflicts of interest for this article.

Open-Access:This article is an open-access article that was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution NonCommercial (CC BYNC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is noncommercial. See: https://creativecommons.org/Licenses/by-nc/4.0/

Country/Territory of origin:China

ORCID number:Liu-Liu Cao 0000-0001-8078-5634; Mei Peng 0000-0001-7809-9440; Xiang Xie 0000-0001-5514-2362;Gong-Quan Chen 0000-0002-4204-6133; Shu-Yan Huang 0000-0002-7050-9553; Jia-Yu Wang 0000-0001-9902-0666; Fan Jiang 0000-0001-8896-2009; Xin-Wu Cui 0000-0003-3890-6660; Christoph F Dietrich 0000-0001-6015-6347.

S-Editor:Fan JR

L-Editor:A

P-Editor:Fan JR