GAO Xiang,ZHANG Xiao-jing

(College of Electrical Engineering and Automation,Shandong University of Science and Technology,Qingdao 266590,China )

The computer keyboard is one of the most common input devices. As an important input device of the computer,it has been playing an irreplaceable role,and has been widely used in microcomputer and a variety of terminal equipments. With the development of the laptop,people need more and more portability,but the size of ordinary keyboard is too large and the usability is not very good. A laser virtual keyboard comes into being which is without entity but could reach real keyboard’s functions. With the developments of science and technology and the continuous upgrading of the human-computer interaction device,the laser virtual keyboard will replace the traditional keyboard gradually and be widely used. It has advantages such as small volume,portability,convenient operation,and accuracy of data input. And it provides people a comfortable way of human-computer interaction.

The core problem of the laser virtual keyboard is to accurately determine the position of the “click”on the virtual keyboard. The paper puts forward a method for laser virtual keyboard key positioning,which adopts infrared laser light reflection and image processing techniques to determine coordinates of the key. Based on the virtual keyboard,the experiment system is established,and the accuracy of the method,the real-time and the stability are verified.

1 Key positioning principle

Laser virtual keyboard consists of infrared camera equipmentA,red laser projectorB,IR laser emitterCand communication system[1]. Structure of the laser virtual keyboard is shown in Fig.1.

Fig.1 Structure of the laser virtual keyboard

Red laser projectorBcan project laser virtual keyboard in any plane. IR laser emitterCcan emit infrared beam level covering the projection area of the virtual keyboard. When people “click” on the keyboard,the fingertips block the infrared beam and a crescent-shaped light spot reflection is formed at the same time,then the “click” is determined. Infrared camera equipmentAcan collect the images of the key operations and store them in the cache. The image processing module with a fixed frequency can extract color images from the cache of video streaming frame by frame and convert them to grayscale images. It's need to select an appropriate threshold for the binarization processing and to identify the position of the crescent-shaped light spot. The position coordinates can be converted into the keyboard plane coordinates. Compared with the keyboard code table,the actual character which is entered into the computer can be identified. The flowchart of the key positioning is shown in Fig.2.

Fig.2 The flowchart of the key positioning

2 Implementation method

2.1 Image acquisition

Firstly,we should open the infrared camera equipment and adjust sharpness and shooting angle,making image acquisition cover the entire projection area of the keyboard. Secondly,people “click” on the virtual keyboard and fingertips block infrared beam,and fingertips’ brightness should be higher than the moon light formed in other areas without launched. Finally,the images are collected through the infrared camera and stored in the video cache as original processing data.

2.2 Image transformation

Image conversion refers to turning color images into grayscale images,and is the basis for the subsequent threshold selection and binarization processing. Color images collected by the standard CMOS sensor with the red(R),green(G) and blue(B) of the three primary colors represent pixel colors. For the need of target recognition,the color images are converted into the grayscale images by floating-point arithmetic which is shown as the Eq.(1)[2].

Gray(i,j)=0.30R(i,j)+0.59G(i,j)+0.11B(i,j),

(1)

whereGray(i,j)represents pixel whose position is (i,j),the gray value corresponding to the range of 0-255;R(i,j),G(i,j),B(i,j)respectively represent color images ofR,G,Bwhose channels in the coordinate are (i,j),namely position of the pixel value.

2.3 Threshold selection and binarization processing

The image binarization processing means to set the pixel gray value to 0 or 255,and it needs to select an appropriate thresholdT. The paper adopts the global dynamic binarization method that is Otsu method[3]. Using a certain threshold divides the gray images into target portion and background portion. The optimal threshold is got when the within-cluster variance is minimum and the between-cluster variance is maximum. The specific methods are as follows：

First,calculating the average gray image. All the pixel gray values of the image can be counted for known images. The average gray valueTavris got by Eq.(2).

Tavr=∑T(i,j)/(width+length),

(2)

wherewidth,lengthis the width and length of the image respectively;T(i,j)is the coordinates for (i,j) of the grayscale value of pixel.

Second,settingTas segmentation threshold of target portion and background portion. The proportionw1of target pixel (whose gray values are greater thanT) of the image can be got by Eq.(3). The average grey value of the target pixelTavr1can be got by Eq.(4).

w1=W1/(width*length),

(3)

Tavr1=∑T(i,j)/W1,T(i,j)>T,

(4)

whereW1is a statistical number of the gray value which is greater thanT. Similarly,the ratiow2which stands for background pixel accounts for the image,and the average grey value of the background pixelTavr2can be got.

Third,findingGwhich arrives at maximum in the Eq.(5)[4]. At this time,the threshold valueTfinalis the best threshold.

G=w1(Tavr1-Tavr)2+w2(Tavr2-Tavr)2.

(5)

After selecting the optimal threshold,grayscale image is binarized for the sake of stressing the position and the shape[5]. According to the gray value of each element of the image which is higher or lower than the threshold value,grayscale image is binarized as the Eq.(6).

(6)

whereD(i,j)represents the coordinates (i,j) of the gray value which is processed by binarization processing. When the condition accords withT(i,j)≥Tfinal,the point pixel is set to 255,otherwise,it is set to 0. The color image is captured by camera equipment,and image conversion and binarization processing are shown in Fig.3.

Fig.3 Dragram of image conversion and binarization processing

2.4 Identification of key placement

The coordinates of acquired targets from the images are built in the image coordinate space,while the projection of the virtual keyboard is built in a physical coordinate space. In order to achieve the planes of corresponding infrared laser and the keyboard,coordinate transformation is required. The image coordinate system is mapped to the physical coordinates. Coordinate transformation can convert the “click” characters to keyboard input characters. We can accomplish the coordinate transformation[6]by Eq.(7).

(7)

When the “click” happens in the keyboard area,position of the crescent-shaped spot can be reflected by fingertips into relative coordinates. Compared with the keyboard code table,the actual coordinates of the button placement can be identified[7]. The keyboard code table is shown in Fig.5.

Fig.4 Diagram of the infrared laser plane

Fig.5 Diagram of the keyboard code table

3 Analysis of experimental verification data

3.1 Experimental design

Indoor lighting condition is less than 600 lx,and the keyboard projector needs a wavelength of 650 nm,the red light laser of 20 mw power. Transmission grating inscribed with the standard keyboard is used to project keyboard image. The keyboard projection size is set as 280 mm×110 mm,and the keyboard projection image distance is 100 mm[8]. The red laser is adopted to avoid disturbing of other light. And the band of light brightness can reach the maximum.

The CMOS sensor has a good effect in capturing images and has high sensitivity. It responds to a wide spectral range that includes the ultraviolet light,the visible light and the infrared waveband. Taking into account the real-time and accuracy of the keyboard operation,the infrared image acquisition system adopts CMOS sensor,that the image resolution is 640×480 pixel and the frame rate is 30 fps[9]. To suppress the interference of ambient light on the system,the infrared filter whose bandwidth is 800-1 000 nm,is installed in the preceding section of the camera.

The point light source of infrared laser emission wavelength of 850 nm is not visible. After modulating the narrow rectangular grating,divergence angle of 120° and linear laser beam of the thickness of 1.5 mm are formed[10]. The linear laser beam is closed to the surface of the projection keyboard and covers the entire projection keyboard flatly.

3.2 Experimental results

In order to verify the accuracy of the key positioning method,the experiments are conducted 5 times and the laser virtual keyboard “click” numbers of each experiment are no less than 100 times. Under normal indoor lighting conditions,the test results are shown in Table 1.

Table 1 Key positioning test results

4 Conclusion

In this paper,we describe key positioning method of the reflection from the infrared laser positioning,image acquisition and processing and coordinate mapping. And the experimental system is established. Experimental results show that the method can successfully achieve good accuracy of key positioning of laser virtual keyboard. The accuracy of the response can fully meet the needs of the input of the computer apparatus. And the system just needs camera equipment,infrared emitter and other simple hardwares. The method of positioning has high reliability and practicality,and its robustness could meet the requirements that the interactive mode of the electronic equipment is simple and convenient.

[1] WANG Zhong-de. The ideas of infrared virtual keyboard design. Computer Engineering,2004,30(6): 189-192.

[2] LIU Qing-xiang,JIANG Tian-fa. A study of translation arithmetic between color image and grey image. Journal of Wuhan University Technology (Transportation Science & Engineering),2003,27(3): 344-346.

[3] Takao N,Shi J,Telegraffiti B S. A camera-projector based remote sketching system with hand-based user interface and automatic session summarization. International Journal of Computer Vision,2003,53(2): 115-120.

[4] LIANG Hua-wei. Direct determination of threshold from bimodal histogram. Pattern Recognition and Artificial Intelligence,2002,15(2): 253-256.

[5] Gonzalez R C,Woods R E. Digital image processing. New Jersey: Prentice Hall,2002.

[6] ZHANG Yu-jin. Image engineering. Beijing: Tsinghua University press,2006.

[7] Sonka M,Hlavac V,Boyle R. Image processing,analysis and machine vision. Beijing: People's Posts and Telecommunications Press,2007.

[8] Hartley R,Zisserman A. Multiple view geometry in computer vision. Oxford: Cambridge University Press,2004.

[9] Konolige K,Augenbraun J,Donaldson N. A low-cost laser distance sensor. International Conference on Robotics and Automation. Pasedena,CA,USA,2008.

[10] Ghosh S,Sarcar S,Sharma M K,et al. Effective virtual keyboard design with size and space adaption. Proceedings of the IEEE Students’ Technology Symposium. Kharagpur,India: IEEE Press,2010.