Junchao TU, Mingang WANG, Liyan ZHANG

College of Mechanical and Electrical Engineering, Nanjing University of Aeronautics and Astronautics, Jiangsu 210016, China

KEYWORDS Calibration;Fabric prepreg placement;Laser materials processing;Laser scanner;Three-dimensional laser marking

Abstract Marking arbitrary three-dimensional (3D) target curves on given objects with curved surface is required in many industrial f ields, such as fabric prepreg placement in composite material part fabrication, product assembly, surface painting for decoration, etc. A shortcut to the solution of this intractable problem is proposed by utilizing a galvanometric laser scanner(GLS) with the aid of a camera. Without using the existing tedious GLS calibration procedures,the proposed method directly establishes a mapping between the 3D coordinates of the laser spots on the object surface and the control voltages of the scanner. A single-hidden layer feedforward neural network (SLFN) is employed to model the mapping. By projecting a dense grid of laser spots on the object to be marked and simultaneously taking only one image, the SLFN model is trained in minutes via a linear solving mechanism. Experiments demonstrate that the trained SLFN model has a good generalization performance for marking 3D target curves. The 3D laser marking errors on experimental objects are less than 0.5 mm. The proposed method is especially suitable for on-site use and can be conveniently extended to multiple GLSs for marking large complex objects.

1. Introduction

In ship building, aircraft manufacturing, and some other industrial f ields, there is an increasing demand for precisely marking operation regions on objects with curved surface to guide a worker/machine fulf illing relative tasks, such as prepreg ply placement in composite material part fabrication,material processing of complex three-dimensional (3D) shape,logo painting for product decoration, parts/components assembling, quality inspection, and so on. Taking the prepreg ply placement in composite material part fabrication as an example,workers in the process are typically in charge of manually placing prepreg plies with different border shapes layer by layer onto specif ied regions of a mould with curved surface.In the traditional way of determining the ply placement regions,each layer needs a physical layup template to be overlaid on the mould for instruction. Building a large amount of real templates is obviously expensive and time-consuming.Although a few commercial laser marking systems based on expensive laser scanners and laser radars are available at present, more economic, f lexible, and concise solutions are still in need.The method proposed in this paper makes use of a galvanometric laser scanner (GLS) together with an off-the-shelf CCD camera to mark 3D target curves on arbitrary known curved surfaces.

Due to the good characteristics on precision, eff iciency,f lexibility, and economy, galvanometric laser scanners are commonly used as the key component of a variety of devices for diverse applications, such as laser machining,1-5laser projection,6,7optical tracking,8-10optical metrology,11,12etc. A typical galvanometric laser scanner consists of two rotatable mirrors driven by two limited-rotation motors respectively.The incoming laser beam is def lected by the mirrors. Consequently, the outgoing direction of the laser beam changes with the mirrors' directions. Since the mirrors' rotation angles, and in turn the outgoing laser beam position and direction, are determined by the control voltages of the motors, most GLS-based applications need a dedicated procedure to accurately establish the relation between the control voltages and the straight line function of the outgoing laser beam. This procedure is known as galvanometric laser scanner calibration, which is the essential problem of underlying applications.

The existing GLS calibration methods can be classif ied into three catalogs,namely,physical-model-based methods,13,14the agent-model-based method,15and universal-model-based methods.16,17Physical-model-based methods construct mathematic models with a set of meaningful parameters to reveal the real status of a device. Due to the complexity of a GLS, various distortion correction mechanisms were put forward to compensate for model errors.18,19These distortion correction methods are applicable only if the object that the laser beam hits on is planar,20or only work for some selected outgoing rays.21As a representative physical-model-based method,Ref.14presents a complicated model containing up to 26 physical parameters to describe more affecting factors in the relationship between the mirror rotation angles and the outgoing laser beam. Even so, not all affecting factors are involved.Excessive optimization parameters usually mean heavy computational burden and high risk of local minima. The agentmodel-based method15adapts the mathematic model of a pinhole camera with distortion to represent a GLS. Since a real GLS does not have an optical center as that of a camera, the agent model has to employ more distortion parameters to ensure the accuracy. Therefore, it faces the same optimization diff iculties as physical-model-based methods do. On top of that, for the problem of marking 3D target curves on curved surfaces, the calibration process of the agent-model-based method is too complex. Universal-model-based methods16establish universal regression models, for instance, artif icial neural networks, to describe the complex relations to be calibrated.The large amount of variables in universal models have no specif ic physical meaning and are usually determined by means of supervised learning from a training data set. This kind of methods are also called data-driven methods. Along this line, Wissel et al.16calibrated a galvanometric triangulation device,in which a GLS and a camera construct a f ixed triangulation structure. They combine 2D control voltages and 2D image coordinates together to serve as the 4D training input and the underlying 3D coordinates of the laser spot in the calibration space as the output label. The established 4Dto-3D mapping is only applicable for the specif ic triangulation setup to measure the shapes of 3D objects. Universal-modelbased methods are more adaptive to different hardware structures and can achieve higher calibration accuracy. However,this type of methods usually takes a long time for acquiring the large amount of training data needed.

The aforementioned methods solve the GLS calibration problem in different ways, but they are inconvenient or even inapplicable for the 3D laser marking problem.The reason lies in two folds. Firstly, in the 3D laser marking scenario, the geometry of the curved surface object as well as the target curve to be marked is known in advance. What we need to do is to eff iciently determine the corresponding voltage values applied to the motors for the given 3D coordinates on the object. This is an inverse problem. The existing calibrated models,no matter in what specif ic forms,are highly nonlinear.As a result,the reverse calculation needs another optimization procedure after the calibration. Some calibration results, for instance, the 4D-to-3D mapping in Ref. 16, cannot even support the reverse calculation at all. Secondly, the calibration data needed for 3D applications is typically captured by projecting a laser spot grid onto a planar board.The board needs to be moved time and again in various positions and poses to cover the working volume as far as possible. Such calibration procedures are time-consuming and not convenient for on-site use.Besides,Diaci et al.22performed laser marking on tilted or curved surfaces based on the tight integration of a 3D laser measurement system and a 3D laser marking system. With the built-in camera and the laser lines from the GLS, the 3D shape of the object to be processed was measured on-site. In the laser marking stage, they simply projected a predef ined 2D binary image with the laser scanner onto the 3D object,using the acquired 3D shape to determine the laser beam focus in the depth direction. In this way, the reverse calculation of the input voltages of the GLS was avoided. However, this 2.5D solution would inevitably result in geometrical distortion for marking complex 3D surfaces just as the authors pointed out in their paper.

In this paper, we propose a very concise way to mark 3D target curves on a given object surface via a GLS. We directly establish the inverse relation between the 3D coordinates of the laser spots on the object surface and the underlying two control voltages of the scanner. This reverse relation is called 3D-to-2D mapping in this context, and is modeled by a single-hidden layer feedforward neural network (SLFN).23Extreme learning machine (ELM)24is applied to eff iciently solve the model. Other than the existing tedious calibration procedures, our calibration procedure needs only to project a grid of laser spots on the standstill object to be marked and take the image. The camera does not need to be moved either in the calibration procedure. It is much easier to implement and more f lexible for industrial applications.

The remaining of the paper is organized as follows. In Section 2, the conf iguration and working mechanism of the proposed 3D laser marking system are brief ly introduced. The method for calibrating the 3D-to-2D mapping is discussed in Section 3. Experiments are included in Section 4 to demonstrate the performance of the proposed method. The paper is concluded in Section 5.

2. System conf iguration and working mechanism

2.1. System conf iguration

As shown in Fig. 1, the proposed 3D laser marking system consists of a laser transmitter, a double-mirror galvanometric scanner, a calibrated CCD camera, a computer, and a control board. As mentioned in Section 1, each mirror is connected with a rotation-limited motor. The computer sends signals to the control board to control the opening or closing of the laser transmitter. In addition, the computer sends voltage values to the control board to control the rotation angles of the dual mirrors.The laser beam emitted from the laser transmitter gets through the two mirrors of the galvanometric scanner. The outgoing laser beam casts on the object and generates a laser spot on the curved surface. By extremely fast changing the rotation angles of the mirrors, the galvanometer produces a fast-moving laser spot on the object. The frequently refreshed trajectory of the fast-moving spot on the curved surface can be observed by human eyes (and the CCD camera) for the vision persistence phenomena. The CCD camera is in charge of 1)aligning the camera and the object and 2)collecting calibration data. The task of the system is making the laser spot moving trajectory precisely mark the target curve contour on the curved surface of a given object.

2.2. System working mechanism

For the convenience of depiction, we f irstly introduce some symbols used in the paper. The curved surface of the object to be marked is denote as Sm, and the target curve contour on Smis denoted as Cm. We use Oc-XcYcZcand Oo-XoYoZoto represent the camera coordinate system and the object coordinate system,respectively.The geometry models of Smand Cmin Oo-XoYoZoare known in advance.Denote the digital voltage value applied to the motor of the f irst mirror as dx,and that of the second mirror as dy.The symbol d represents the two-dimensional(2D)digital control volt-

Before starting the marking procedure, put the GLS, the camera, and the object to be marked in proper relative positions so that the object Smis in the working volume of the GLS and in the camera's f ield of view.There are three sequential stages involved:1)the coordinate systems alignment,2)the 3D-to-2D mapping calibration,and 3)the target curve contour marking.

Fig. 1 System conf iguration.

1) The coordinate systems alignment.In this stage,it needs only to take an image of the visual feature points laid around Smby the camera. The 3D coordinates of these feature points in Oo-XoYoZoare known in advance.According to the image coordinates and the corresponding 3D coordinates of the feature points, the transform matrix between Oc-XcYcZcand Oo-XoYoZois computed based on the well-known perspective-n-point(PnP) theory.25,26Then the geometric models of Smand Cmare transformed to Oc-XcYcZc. In the rest of the paper, Smand Cmare in Oc-XcYcZcunless otherwise specif ied.

2) The 3D-to-2D mapping calibration. In this stage, a set of control voltages( i=1,2,···,N) is sent to control the GLS to project a dense grid of laser spots onto the curved surface Sm. The camera record the image of Smwith the laser spots on. According toand the extracted image coordinatesof the laser spots on Sm, we establish the relation between the 3D coordinates P on Smand the input control voltage v, i.e., the reverse 3D-to-2D mapping, which is denoted as M:P →d.Since it requires only the image of Smwith laser spots on and since our algorithm for solving M:P →d is very fast,this stage can be f inished in minutes.In an actual working procedure,the camera can be taken away after this stage, since its task has been completed.

3) The target contour marking. Having established M:P →d,the target marking stage is straight forward.Substituting the discrete point sequence(k =1,2,···,l) of Cminto the established mapping M:P →d, we can directly calculate the digital signalscorresponding to.By inputting these signals into the control board, the GLS will generate a 3D target contour on the curved surface Sm.

From the above descriptions, we can see that Stage 1 is a well-studied procedure for aligning a calibrated camera and a known object, and Stage 3 is a straightforward calculation procedure.The essential of the system is the 3D-to-2D calibration in Stage 2, which will be detailed in the next section.

3. Modeling and calibrating of 3D-to-2D reverse mapping

3.1. Regression model

We model the 3D-to-2D reverse mapping M:P →d as a regression problem in this paper.Considering the strong capability of artif icial neural networks for non-linear mapping approximation,we construct a single-hidden layer feedforward neural network (SLFN) as shown in Fig. 2 to represent the reverse mapping between the 3D coordinates P= X,Y,Z[ ]Ton Smand the 2D voltage signalfor controlling the GLS. There are three neurons in the input layer, which respectively are the X,Y,and Z components of the 3D coordinates P. The output layer contains two neurons, which are respectively the voltage signals dxand dy. Denote the number of neurons in the hidden layer as L.

Fig. 2 SLFN structure of the reverse 3D-to-2D mapping.

The regression model of M:P →d is formulated as

The model in Eq. (1) is trained with a training set. The following two subsections describe the training method in details.

3.2. Generating a training set

As described in Section 2.2, an image of Smwith a grid of N laser spots projected on is captured in the 3D-to-2D mapping calibration stage. The image coordinatesof the laser spot centers are f irstly extracted from the captured image. Then according to the pin-hole imaging theory (to clarify the depiction, here we ignore the camera distortion, which can be compensated for a calibrated camera), we have

where the 3×3 matrix A is the camera intrinsic parameter matrix, P is a 3D point in the camera coordinate system Oc-XcYcZc,= u,v,1[ ]Tis the homogeneous coordinates of the image point of P, and λ is the scale factor. Given, we can obtain the parametric function r(t )in Oc-XcYcZcof the straight line connecting the camera coordinate system origin and the image pointas

where the scalar t is the parameter of the straight line.The 3D coordinatescorresponding to the image pointon Smcan be achieved by intersecting the straight line r(t)with the geometric model of Sm. In this way, we achieve the 3D coordinates(i=1,2,···,N),as shown in Fig.3.By associating(i=1,2,···,N) with the input control voltagesthe training data setis fully achieved.

3.3. Solving the regression model

Feedforward neural networks are extensively used in many applications, and gradient-descent-based methods27are commonly used for training the networks. In these methods, all the parameters wj,bj,and βj(j=1,2,···,L)of the feedforward networks need to be tuned iteratively in many steps, and the gradient computation in each step is typically timeconsuming. Therefore, gradient descent-based learning methods are generally criticized for being slow or easily converging to local minima.To eff iciently establish the 3D-to-2D mapping M:P →d, we incorporate ELM23,24to solve the model formulated in Eq. (1).

Given N arbitrary distinct sampleswhere∈R3are the input andare the output, the SLFN with L hidden neurons in Eq. (1) should satisfy

Eqs. (5) can be rewritten compactly as

where

It has been rigorously proven in Ref.23 that an SLFN with randomly generated wjand bj(j=1,2,···,L) has the capability to approximate the f inite set of training samples. More specif ically,given any small positive value ε ＞0 and randomly chosen wjand bj,if only the activation function g(·)is inf initely differentiable, there exist L ≤N hidden nodes, so that for N arbitrary distinct samples, ‖Hw,b,Pβ-D‖＜ε holds.

Fig. 3 Generating a training set.

Upon the above conclusion, we randomly assign input weights wj(j=1,2,···,L) and hidden layer biases bj(j=1,2,···,L). Then, the hidden layer output matrix Hw,b,Pin Eq. (7) is fully determined, and the model in Eq. (8)can be simply considered as a linear system. The output weights in β is analytically determined by

The above simple procedure for solving the regression model contains no non-linear optimization. Instead, it only needs to solve a linear system in a closed form. Our experiments demonstrate that the 3D-to-2D mapping M:P →d established in this way has good performance.

4. Experiments

In order to verify the validity and practicability of the proposed method, experiments on real curved surfaces were performed. Fig. 4 illustrates the 3D laser marking prototype we built. The prototype makes use of an economic 520-nm semiconductor laser and a TSH8050A/D Century Sunny galvanometric. Both the laser transmitter and the galvanometric scanning head are controlled by a GT-400-Scan control board.

Fig. 4 3D laser marking prototype for the experiments.

The software for realizing the three working stages, namely,the coordinate system alignment, the M:P →d calibration,and the target curve marking, is installed in a personal computer with a 2.0-GHz CPU and a 4G RAM.

4.1. Calibration experiment

The f irst object used in the experiment is illustrated in Fig. 5(a).It is a forming mold of a sheet metal part of an unmanned airplane. There are scribed curves on the mold for indicating the trimming border of the sheet metal part.

In this laboratorial experiment, a number of visual feature points (the centers of the white circles as shown in Fig. 5(a))were pasted on the border of the mold for aligning the coordinate system. The 3D coordinates of the visual feature points,the 3D cloud points of the mold surface, and the 3D points on the scribed curves were simultaneously measured in advance by the commercial 3D measuring system ATOS?.In this way, the 3D coordinates of the feature points and the mold surface together with the scribed curves are in the same coordinate system Oo-XoYoZo. The mold surface and the scribed curves established with the reverse modeling method were taken as the theoretical model as shown in Fig. 5 (b).In the calibration stage, the transformation matrix from the mold coordinate system Oo-XoYoZoto the camera coordinate system Oc-XcYcZcwas calculated by using only one image of the feature points and the PnP theory25,26mentioned in Section 2.2.Then,the geometric models of the mold and the scribed curves were transformed to Oc-XcYcZc. In a real manufacturing process,the mold is typically f ixed with a positioning jig. By setting technical holes/bosses on known positions of the jig, the relative geometric relationship between the holes/bosses and the mold can be strictly guaranteed by manufacturing process control. The centers of these technical holes/bosses can be used as the visual feature points for system alignment.

We then input 17,920 pairs of voltage signals

Fig. 5 Object with scribed curves for the experiment.

The focus of the experiment in this subsection is to verify the calibration accuracy. For this purpose, 900 pairs of test voltage signalscontrolled the GLS to project 900 laser spots on the mold surface. To ensure the independency of the test, no test voltage signals were the same as the calibration voltage signals. It is worth pointing out that although in actual working scenarios the camera can be moved away after the mapping M:P →d has been calibrated, we kept the camera in its position in the experiment for the following accuracy verif ication. The 3D coordinatesof these laser spot centers were calculated from the image coordinatesagain by using the method proposed in Section 3.2. We then substitutedinto the calibrated M:P →d to estimate the corresponding voltage signals. The difference between the actual input signaland the estimated input signalis def ined as

Fig.6 Error distribution of estimated control voltages dx and dy by our method.

According to the XY2-100 protocol of the GLS we used,the voltage signalsare dimensionless quantities.The ranges of dxand dyof the scanner used are -32768 to 32767, respectively, and the rotation angle ranges of the mirrors are -12.5° to 12.5°, respectively. In the experiment, the maximums ofand(i=1,2,···,900) are 17.8 and -12.4, respectively, which correspond to 0.007° and-0.0048° rotation angle errors for the two GLS mirrors,respectively. Experimental results of the 3D projection error in terms of space distances are included in the next subsection.

For comparison purpose,a physical-model-based method14and a data-driven method17are realized to calibrate the GLS respectively. The physical-model-based method establishes the mapping M:P →l between the GLS input signals d and the corresponding outgoing laser beam l,in terms of real structural parameters via nonlinear optimization, while the datadriven method establishes the mapping M:P →l by training a neural network. With the two calibration results, the GLS marked the 3D target curves on the surface shown in Fig. 5(b) respectively. According to(i=1,2,···,900)and the calibration results of the two comparison methods, we obtained the corresponding voltage signalsby solving

for each of the comparison methods. Here, Di(d) is the distance from the 3D pointto the spatial laser beam l corresponding to the digital voltage d.According to Eq.(9),we further calculated the digital voltage errors(i=1,2,···,900)for the two comparison methods respectively. Then, the root mean squared errors (RMSEs)andwere used to indicate the calibration accuracies of the methods taking part in the comparison.Results are listed in Table 1,where Tcalibrepresents the approximate duration of the whole calibration procedure, and Tcalcurepresents the running time of calculating the digital.As shown in Table 1,the accuracy of the proposed method is superior to that of the physical-model-based method and similar to that of the existing data-driven method. More importantly, the proposed method is more eff icient than the other approaches for marking 3D target curves. The reason lies in twofold.Firstly,since the proposed method directly constructs the mapping M:P →d, it does not need to solve the nonlinear optimization in Eq.(10) to calculate the control signals for the given 3D points.Secondly,the sample data needed for calibration is much less than that of the other approaches,since our method only needs to get the 3D coordinates of the laser spots on the workpiece surface with only one image,whereas the other two methods need the 3D coordinates of a serial of points on each outgoing laser beam with the aid of an auxiliary calibration board.

4.2. 3D curve marking experiments

In this section, we conducted curve contour marking experiments to validate our method. The scribed curve highlighted in Fig. 5(b) is selected as the target contour to be marked.The theoretical target curve is f irstly discretized into 145 sequential points and transformed into the camera coordinate system Oc-XcYcZc, denoted as. By using the calibration result M:P →d achieved in Section 4.1,the corresponding voltage signals(k=1,2,···,145)are obtained.These digital signals are input into the control board by the computer, and then laser marking results can be observed on the mold surface.The GLS has two scan modes: point scan and linear interpolation scan. In the point scan mode,the laser transmitter is turned off between inputting any two adjacent pairs of digital signals, and the laser marking trajectory is pointwise as shown in Fig. 7(a),whereas in the linear interpolation scan mode, the laser transmitter keeps turned on all the time,and the marking trajectory is continuous as illustrated in Fig. 7(b). With human visual observation, the laser marking trajectories perfectly overlay the physical scribed curve on the mold. To further quantitatively evaluate the 3D laser marking accuracy, the 3D coordinates of the 145 laser spots actually projected on the mold surface, denoted as, are obtained by using the method in Section 3.2. The Euclidean distancebetweenandis calculated and illustrated in Fig. 8 (a). The average marking error isthe maximum of, and the standard deviationAn intuitive comparison between the theoretical target contour and the actual laser marking contour, i.e., the linear interpolation of the points(k=1,2,···,145), is shown in Fig. 8 (b). We can see that the two contours closely coincide together.

The above experiment implies two potential applications of the proposed method.On one hand,by precisely projecting the target contour on the mold, one can visually determine if the physically scribed curve contour is correct. This is actually a way of inspection. On the other hand, the laser marking contour on the mold can directly replace the physically scribed contour to guide an operator for trimming the sheet metal part. Therefore, it is also a way of visual guidance for working.

Table 1 Performances of three calibration methods.

Fig. 7 Real laser marking contours.

Fig. 8 Evaluation of the marking accuracy.

Fig. 9 Another 3D curve marking experiment.

Fig. 9 demonstrates another 3D target curve marking experiment on a curved rectangular sheet metal. In this experiment, the theoretical 3D surface model (Fig. 9(a)) as well as the system alignment was achieved in the same way as that used for the forming mold. On the reversely established 3D surface model, we designed the target curve ‘‘A203” in the computer, which is composed of 181 discrete points. We projected 19,200 laser spots on the object surface and took an image to calibrate the mapping M:P →d with the method in Section 3. Based on the calibration result, the real marking result on the physical sheet metal is shown in Fig. 9(b). With the same evaluation method as that used in the last experiment, we obtained the average marking errorthe maximum ofis 0.49 mm, and the standard deviationThis experiment demonstrates once again the good performance of the proposed method.

5. Conclusions

A concise method for marking arbitrary target contours on 3D curved surfaces is presented in this paper.Based on a principle analysis and experimental results, we draw the following conclusions:

1) By directly establishing the reverse mapping M:P →d,we can avoid extra nonlinear optimization for the reverse computation involved in the 3D laser marking problem. In addition, the calibration procedure needs only to project a grid of laser spots on the object to be marked and take the image. This greatly simplif ies the calibration process and calibration computation.Experimental results show that the entire calibration time can be reduced to approximately one-third of the time used for existing methods.

2) The SLFN-based regression model exhibits a good generalization performance for representing the relationship between the laser spot P on the object surface and the input signal d.In the experiments,the 3D laser marking errors are less than 0.5 mm. The ELM (Extreme Learning Machine)proves to be an effective and eff icient way for solving the regression model.By using the ELM,the regression model can be converted to a linear system and solved in seconds.

3) The relative pose between the GLS and the camera doesn't need to be f ixed and calibrated in advance,which makes the system more f lexible and more suitable for on-site use.

It is worth of notice that due to the limitation of the GLS scanning speed,one GLS may be insuff icient for marking large and complex objects. This situation requires multiple GLSs working together, each in charge of a sub-region of the object surface. With the proposed method, when the 3D-to-2D mapping of a GLS has been calibrated with the aid of a camera,the camera can be moved to another proper position for calibrating another GLS. After the 3D-to-2D mappings of all the GLSs have been successively determined, the camera can be moved away from the working f ield. Therefore, the proposed method can be conveniently extended to multiple GLSs. In addition, by increasing the power of the laser, the proposed method also applies to 3D laser surface processing, 3D laser engraving, and 3D laser cutting.

Acknowledgement

This study was partly supported by the National Natural Science Foundation of China (No. 51575276).