,This is the second block}

其中nA、nB分別表示節(jié)點(diǎn)A、B的深度。

3 實(shí)驗(yàn)結(jié)果分析

本文從新浪、網(wǎng)易、搜狐、騰訊等大型門戶網(wǎng)站以及多家各類型網(wǎng)站中抽取了1 000個(gè)網(wǎng)頁作為測(cè)試數(shù)據(jù)，采用基于網(wǎng)頁DOM樹節(jié)點(diǎn)路徑相似度的正文抽取方法進(jìn)行實(shí)驗(yàn)，去噪結(jié)果和正文抽取結(jié)果如表1所示。

表1 本文方法的正文抽取實(shí)驗(yàn)結(jié)果

從表1的統(tǒng)計(jì)結(jié)果可以看出，有97.6%的網(wǎng)頁清洗掉了大部分的噪聲并且完整保留了網(wǎng)頁中的有效信息；對(duì)于新浪、網(wǎng)易等門戶網(wǎng)站的抽取結(jié)果較好，都有90%以上的準(zhǔn)確率和95%以上的召回率；對(duì)于其他不同結(jié)構(gòu)的網(wǎng)站，本文的正文抽取方法也都能適用，很好地實(shí)現(xiàn)了網(wǎng)頁正文抽取的工作，并且有著較高的準(zhǔn)確率和召回率。

為了驗(yàn)證本文方法的有效性，以上述的1 000個(gè)網(wǎng)頁作為樣本，將本文方法與BTE、DSC、FE、LQF、CCB等算法進(jìn)行對(duì)比實(shí)驗(yàn),實(shí)驗(yàn)結(jié)果如表2所示。

表2 各種算法對(duì)比結(jié)果

由表2可以看出，本文提出的方法相對(duì)于現(xiàn)有的統(tǒng)計(jì)方法有更好的準(zhǔn)確率和召回率。

互聯(lián)網(wǎng)的發(fā)展為用戶帶來了一個(gè)包含豐富信息的巨型數(shù)據(jù)庫，但是如何識(shí)別其中的有效數(shù)據(jù)是應(yīng)用的關(guān)鍵。本文的正文抽取方法利用網(wǎng)頁DOM樹節(jié)點(diǎn)路徑相似的特點(diǎn)實(shí)現(xiàn)正文抽取，為之后的數(shù)據(jù)分類、分析等工作奠定了基礎(chǔ)。

4 結(jié)論

本文根據(jù)新聞?wù)膬?nèi)容在網(wǎng)頁中相對(duì)集中且同網(wǎng)站的新聞頁面有相同模板的特點(diǎn)，提出基于網(wǎng)頁DOM樹節(jié)點(diǎn)路徑相似度的正文抽取方法，先用正則表達(dá)式刪除網(wǎng)頁源代碼中與正文內(nèi)容無關(guān)的代碼，然后將得到的網(wǎng)頁轉(zhuǎn)換為DOM樹，再將目標(biāo)網(wǎng)頁的DOM樹與另外兩個(gè)網(wǎng)頁的DOM樹進(jìn)行對(duì)比去除噪聲，最后，根據(jù)節(jié)點(diǎn)路徑相似度來抽取正文內(nèi)容。該方法對(duì)來自不同網(wǎng)站的數(shù)據(jù)能夠快速、準(zhǔn)確地抽取正文內(nèi)容，適用于結(jié)構(gòu)變化不大的網(wǎng)頁，但是對(duì)正文內(nèi)容較少的網(wǎng)頁抽取效果仍有待提高。下一步主要工作是加入內(nèi)容節(jié)點(diǎn)與標(biāo)題節(jié)點(diǎn)的路徑之間的距離判斷節(jié)點(diǎn)是否為正文，以提高算法的準(zhǔn)確度。

[1] KUSHMERICK N， WELD D S， DOORENBOS R.Wrapper induction for information extraction[C].IJCAI 1997: Proceedings of the 1997 International Joint Conference on Artificial Intelligence，1997:729-737.

[2] FU L,MENG Y,XIA Y J， et al.Web content extraction based on webpage layout analysis[C].ITCS 2010: Proceedings of the 2010 Second International Conference on Information Technology and Computer Science,2010: 40-43.

[3] CAI D,YU S P,WEN J R,et al.VIPS: a vision based on page segmentation algorithm[R].Microsoft Co.,Tech.Report,2003.

[4] WANG J Q,CHEN Q C,WANG X L,et al.Basic semantic units based web page content extraction[C].SMC 2008: Proceedings of the 2008 IEEE International Conference on Systems,Man and Cybernetics,Piscataway，NJ: IEEE Press,2008:1489-1494.

[5] UZUN E,AGUN H V,YERLIKAYA T.Web content extraction by using decision tree learning[C].SIU 2012: Signal Processing and Communications Applications Conference,2012: 1-4.

[6] PAN D H,QIUS G,YIN D W.Web page content extraction method based on link density and statistic[C].WiCOM 2008: Wireless Communications,Networking and Mobile Computing,Dalian,China,IEEE Press,2008:1-4.

[7] REIS D C,GOLGHER P B.Automatic web news extraction using tree edit distance[C].Proc.WWW 2004: The 13th International Conference on World Wide Web,New York: ACM,2004: 502-511.

[8] FINN A,KUSHMERICK N,SMYTH B.Fact or fiction: Con-tent classification for digital libraries[C].Proc of the 2nd DELOS Network of Excellence Workshop on Personalization and Recommender Systems in Digital Libraries.Dublin,Ireland,2001: 1-6.

[9] PINTO D,BRANSTEIN M,COLEMAN R,et al.QuASM: A system for question answering using semi-structured data[C].Proc of the 2nd ACM/ IEEE-CS Joint Conference on Digital Libraries.Portland,USA,2002: 46-55.

[10] MANTRATZIS C,ORGUN M,CASSIDY S.Separating XHTML content from navigation clutter using DOM-structure block analysis[C].Proc of the 16th ACM Conference on Hypertext and Hypermedia， Salzburg,Austria,2005: 145-147.

[11] DEBNATH S,MITRA P,GILES C L.Automatic extraction of informative blocks from webpages[C].Proc of the ACM Symposium on Applied Computing， SantaFe,USA,2005: 1722-1726.

[12] GOTTRON T.Content code blurring: A new approach to content extraction[C].Proc of the 19th International Conference on Database and Expert Systems Applications， Turin,Italy,2008: 29-33.

[13] 劉利,戴齊,尹紅風(fēng),等.基于多特征融合的網(wǎng)頁正文信息抽取[J].計(jì)算機(jī)應(yīng)用與軟件,2014,31(7):47-49.

[14] 王利,劉宗田,王燕華,等.基于內(nèi)容相似度的網(wǎng)頁正文提取[J].計(jì)算機(jī)工程,2010,36(6):102-104.

[15] YI L,LIU B,LI X.Eliminating noise information in web pages for data mining[C].SIGKDD 2003: Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining， New York: ACM,2003:296-305．

[16] GIBSON D,PUNERA K,TOMKINS A.The volume and evolution of web page templates[C].Proc.WWW 2005: Special Interest Tracks and Posters of the 14th International Conference on World Wide Web,New York: ACM,2005:830-839．

[17] WANG J Y,LOCHOVSKY F H.Data-rich section extraction from HTML pages[C].WISE 2002: Proceedings of the 3rd International Conference on Web Information Systems Engineering (Workshops)， Los Alamitos,CA: IEEE Computer Society,2002: 313-322．

Content extraction based on the similarity of the Web pages′ DOM tree nodes path

Pan Xinyu1,Chen Changfu2,Liu Rong1,Wang Meiqing1

(1.College of Mathematics and Computer Science,Fuzhou University,Fuzhou 350108,China; 2.Fujian Ecallcen Information Technology Company Limited,Fuzhou 350000,China)

Due to the problem that the low efficiency and high cost of extracting information by human,according to the observation of large amount of Web pages’structure,the content extraction method based on the similarity of web pages’ DOM tree node’s path was proposed.It removed noise and got the main body of the Web page as the Web pages in the same website had the same structure,then combined the similarity of the path of content nodes in the DOM tree to extract content.Through the experiments of 1 000 Web pages from different Chinese news Websites,the results show that this method can remove most noise and maintain the integrity of the content for 97.6% of all Web pages,it has 93.30% precision rate and 95.59% recall rate,and it has good adaptability for different types of Web pages.

DOM tree; information extraction; HTML tag; Web denoising; content extraction

TP301.6

A DOI：10.19358/j.issn.1674-7720.2016.19.022

潘心宇，陳長(zhǎng)福，劉蓉，等.基于網(wǎng)頁DOM樹節(jié)點(diǎn)路徑相似度的正文抽取[J].微型機(jī)與應(yīng)用，2016,35(19)：74-77.

2016-05-13)

潘心宇(1992-)，男，碩士研究生，主要研究方向：數(shù)據(jù)挖掘、模式識(shí)別。

陳長(zhǎng)福(1974-)，男，學(xué)士，主要研究方向：網(wǎng)絡(luò)信息挖掘、信息分類。

劉蓉(1972-)，通信作者，女，碩士，講師，主要研究方向：數(shù)值計(jì)算。E-mail:liu_r@fzu.edu.cn。