Skip to main content

Vision-Guided Robotics and Hand-Eye Calibration Explained

A vision-guided robot uses a camera to find a part and then moves to it, which works only if the robot knows exactly where the camera is. UTEC Industrial designs, engineers, machines, fabricates, and installs custom material handling systems for aerospace and heavy industry from its Spokane Valley, WA facility, integrating Allen-Bradley PLC and motion control with in-house CNC machining, heat treating, and stress relief. This article explains how that relationship is found by hand-eye calibration: where the camera is mounted, what the AX=XB equation means, how many robot poses a solution needs, which solution methods exist, and how the result is verified and kept valid in production. In the build chain, design → engineering → parts machining → fabrication → assembly → weld fatigue → stress relief → drives → controls → tuning → monitoring, calibration is a tuning step, but it depends on the machined and fabricated mounts that hold the camera, the target, and the tooling.

What does a vision-guided robot need to know before it can use a camera?​

A camera reports a part's position in pixels; a robot moves in its own coordinate frame. Getting from one to the other takes a chain of transformations, and every link has to be known:

  1. Camera intrinsics. A model of the lens and sensor that turns pixel positions into geometric rays. Zhang's planar-pattern method, which the category overview on machine vision in material handling describes, is one way to find it.
  2. Target-to-camera pose. Where a calibration pattern sits relative to the camera, found from one image by a perspective-n-point (PnP) solution once the intrinsics are known.
  3. Gripper-to-base pose. Where the robot's flange or gripper is relative to the robot base, read from the robot's own kinematics at each pose.
  4. The hand-eye transform. Where the camera sits relative to the gripper, or relative to the robot base for a fixed camera. This is the unknown that hand-eye calibration solves.

Horaud and Dornaika define the problem this way: hand-eye calibration determines the relationship between a sensor mounted on the robot hand and the hand itself, and it is important in at least two types of tasks, mapping sensor-centered measurements into the robot's workspace frame and moving the sensor precisely. As engineering reasoning, the failure mode when the hand-eye transform is wrong is a robot that picks every part with the same offset, even though the images look correct (Horaud and Dornaika 1995, IJRR 14-3; OpenCV 4.13.0, calib3d, cv::calibrateHandEye; Zhang 2000, IEEE TPAMI 22-11).

Should the camera ride on the robot or stand fixed in the cell?​

The two layouts have standard names. With eye-in-hand, the camera is mounted on the gripper and moves with the robot. With eye-to-hand, the camera is fixed in the cell and the calibration pattern is carried on the end-effector. FANUC's operator's manual for its 2D camera vision application describes each layout:

  • Fixed camera. It always images the same place from the same distance, and it can detect other parts while the robot transfers a workpiece, so the cycle time can be shortened. The manual calls for a camera stand strong enough that the camera does not vibrate.
  • Robot-mounted camera. It is mounted on the robot's final axis and can measure different places as the robot moves, and the vision system calculates the part position from the robot's movement. Its cable moves with the robot and has to be routed so it does not tangle.

The rest of this paragraph is engineering reasoning. Those trade-offs map onto handling applications. A fixed camera over an infeed conveyor suits a steady stream of castings or machined blanks, because imaging overlaps robot motion. A robot-mounted camera suits a large part, such as an aerospace panel or a fabricated frame, where the robot has to look at several features that one fixed view cannot cover. The failure modes follow the layout too: a fixed stand bolted near a press or a vibrating drive moves the camera, and a robot-mounted camera's cable fatigues at the wrist if it is not routed for the robot's full range of motion.

UTEC Industrial integrates FANUC robotic cells, including vision, with a FANUC design and engineering partner (OpenCV 4.13.0, calib3d, cv::calibrateHandEye; FANUC B-83914EN-2/01, Introduction §2.1–§2.2).

What does the AX=XB hand-eye equation mean?​

Hand-eye calibration is usually written as AX = XB, where each term is a rigid-body transformation, a rotation plus a translation:

  • A is the motion of the robot's wrist between two poses, known from the robot's kinematics.
  • B is the resulting displacement of the camera between the same two poses.
  • X is the unknown position of the camera relative to the wrist.

Shiu and Ahmad showed how the problem arises: the camera's pose relative to the wrist is found by moving the robot and observing the resulting motion of the camera. Because A and B are the same physical motion seen from two places, X is the transformation that makes them agree. Their analysis also shows why one move is never enough. If the angle of rotation of A is neither 0 nor π radians, the solution from a single motion still has one rotational and one translational degree of freedom, and to solve for X uniquely two arm movements are needed.

Park and Martin framed AX = XB as the equation that arises when calibrating wrist-mounted robotic sensors. They gave a closed-form exact solution, using Lie theory, and a closed-form least-squares solution for the practical case where A and B are noisy. Horaud and Dornaika kept the classic AX = XB formulation and added an alternative, written MY = M′YB, which avoids making the camera's intrinsic and extrinsic parameters explicit (Shiu and Ahmad 1989, IEEE Transactions on Robotics and Automation 5-1; Park and Martin 1994, IEEE Transactions on Robotics and Automation 10-5; Horaud and Dornaika 1995, IJRR 14-3).

How many robot poses does hand-eye calibration need, and how should they be chosen?​

The OpenCV library's calib3d documentation states the minimum: 2 motions with non-parallel rotation axes, which means at least 3 poses, and it strongly recommends using many more. That matches Shiu and Ahmad's result that one motion leaves the solution undetermined. The requirement for non-parallel rotation axes has a practical consequence: a pose set in which the robot only translates, or rotates about one axis, does not meet the documented minimum however many poses it contains.

The procedure itself is set out in the same documentation:

  1. Fix a calibration pattern so it does not move.
  2. Move the gripper to a series of poses.
  3. At each pose, record the gripper-to-base transformation from the robot's kinematics and compute the target-to-camera transformation from the image.
  4. Solve AX = XB from the pairs of motions.

The pose-choice guidance in the rest of this paragraph is engineering reasoning that follows from the documented minimum rather than a sourced rule. Tilt the camera about different axes from pose to pose. Keep the whole pattern in view and in focus at every pose. Spread the poses over the volume where the robot will actually pick or place parts, so the calibration is exercised where it is used. Avoid moving the pattern between poses, because the method assumes it is static (OpenCV 4.13.0, calib3d, cv::calibrateHandEye Note; Shiu and Ahmad 1989, IEEE Transactions on Robotics and Automation 5-1).

Which methods solve the hand-eye equation, and does the choice matter?​

The calib3d documentation sorts the methods it implements into two approaches:

  • Separable methods estimate the rotation part of X first and then the translation. Tsai and Lenz, Park and Martin, and Horaud and Dornaika are in this group.
  • Simultaneous methods solve rotation and translation together. Andreff and co-authors and Daniilidis are in this group.

Daniilidis argued for a simultaneous solution. He noted that the many algorithms proposed for hand-eye calibration do not treat relative position and orientation in a unified way; his formulation represents each motion as a dual quaternion, the algebraic counterpart of a screw motion, and solves rotation and translation at the same time with a singular value decomposition (SVD). He evaluated it on stereo reconstruction and camera positioning. Tsai and Lenz's method, which the category overview describes with its timing figures, is the default method of the calib3d hand-eye function and sits in the separable group.

The sources compare methods but do not settle the choice for a production cell. Horaud and Dornaika's stability analysis compared their closed-form method, their nonlinear simultaneous method, and Tsai and Lenz's linear method, and concluded that the simultaneous nonlinear method seems the least sensitive to noise and measurement errors. Daniilidis reports his approach outperforming two other methods in his experiments. The next two sentences are engineering reasoning. A practical check follows from having several methods in one library: solve the same data with a separable and a simultaneous method. Close agreement suggests the poses were well chosen, and a large disagreement flags a weak pose set, a bad pose pair, or a moved target, before any production part is picked. This article gives no accuracy figures for any method, because none of the sources cited here states one that applies to a production cell (OpenCV 4.13.0, calib3d, cv::calibrateHandEye; Daniilidis 1999, IJRR 18-3; Horaud and Dornaika 1995, IJRR 14-3; Tsai and Lenz 1989, IEEE Transactions on Robotics and Automation 5-3; Park and Martin 1994, IEEE Transactions on Robotics and Automation 10-5).

How is a fixed camera calibrated when the target rides on the robot?​

For the eye-to-hand layout, the roles swap. The camera is static, the calibration pattern is carried on the end-effector, and the unknown is the camera's pose relative to the robot base instead of relative to the gripper. The calib3d documentation covers this case with the same hand-eye function, given the suitable transformations as inputs. The same documentation also offers a combined robot-world and hand-eye routine, which by default uses Shah's method. It is described for a camera mounted on the robot's end-effector viewing a static calibration target, whose frame serves as the world frame, and it estimates the robot-base-to-world and gripper-to-camera transformations in one calibration. That suits a cell in which the robot has to be tied to a fixed cell frame as well as to its camera.

Two design points follow for a fixed camera, as engineering reasoning from those procedures:

  • The target must be fixed to the tool rigidly. A pattern plate that can shift on the end-effector changes between poses and breaks the static-target assumption. Doweled or keyed mounting, not a single clamp, is the remedy this reasoning points to.
  • The target must be flat and stable. Zhang's intrinsic method relies on a planar pattern, and a warped or thermally unstable plate adds error to both steps.

A related failure mode, also engineering reasoning, is a calibration tool swapped for the production gripper without re-teaching the gripper's tool frame. The camera's calibration is still correct, but the robot's flange-to-part relationship has changed, so the picks are offset until the tool frame is re-taught (OpenCV 4.13.0, calib3d, cv::calibrateHandEye and cv::calibrateRobotWorldHandEye; Zhang 2000, IEEE TPAMI 22-11).

How does a FANUC robot calibrate its camera and apply a vision offset?​

FANUC's operator's manual for its 2D camera vision application lists the parts of the system: the camera and lens, the camera cable, lighting equipment, and a camera multiplexer used if needed; its basic-configuration figure shows the camera cabled to the robot controller. It says there are three methods to calibrate a camera: grid pattern calibration with a fixed camera, grid pattern calibration with a robot-mounted camera, and robot-generated grid calibration.

  • Grid pattern calibration, which the manual calls a general-purpose method, uses a calibration grid fixture, and the manual gives it a fixed-camera version and a robot-mounted-camera version. A robot-mounted camera needs this method.
  • Robot-generated grid calibration has the robot move a target, mounted on the end-of-arm tooling, through the camera's field of view to build a virtual grid. It is for fixed cameras only. Using two planes, it finds the camera position and the lens focal distance.

The manual also describes two kinds of robot position offset that use the result:

  • Fixed frame offset. The camera finds the part on the table, and the robot's positions are offset so the robot works on the part where it actually is.
  • Tool offset. The camera finds the part already held in the gripper, and the robot's positions are offset so the part is placed correctly.

For handling, the tool offset is the one that catches a part that shifted in the gripper, such as a casting picked from a rough locating nest, before it is loaded into a machine fixture; that application is engineering reasoning from the manual's definitions. UTEC Industrial delivers this kind of FANUC vision integration with a FANUC design and engineering partner (FANUC B-83914EN-2/01, Introduction §2.1–§2.2 and §2.4; Know-How Ch. 2 §2.1–§2.3).

What mechanical design decisions keep a hand-eye calibration valid?​

A calibration records a geometric relationship between parts of the machine, so it is only as durable as the mechanics that hold that relationship. The chain links are:

  • Design and engineering decide the camera layout, the working distance, and where the calibration target lives when it is not in use.
  • Parts machining makes the camera bracket, the dowel holes that let a bracket come off and go back in the same place, the target plate, and the tool flange. A bracket that locates on dowels can be replaced without recalibration only if its locating features are held to a tolerance tighter than the pick tolerance; the guide to machining tolerances covers how to put that on a drawing.
  • Fabrication, weld fatigue, and stress relief decide whether the camera stand or robot riser stays put. A fixed camera needs a stand that does not vibrate, as described above; a welded stand that relaxes after installation moves the camera slowly, and every pick drifts with it.
  • Assembly sets the tool and camera on the robot and records the tool frame the calibration assumes.

These mappings are engineering reasoning placing the sources' requirements, a static pattern, a stand that does not vibrate, and a planar target, onto the build chain. UTEC Industrial machines brackets, fixtures, and tool interfaces to tolerances as tight as ±0.001 in and stress-relieves welded frames before final machining, the mechanical base a calibration rests on (OpenCV 4.13.0, calib3d, cv::calibrateHandEye; Zhang 2000, IEEE TPAMI 22-11).

What goes wrong with hand-eye calibration in a working cell?​

The category overview covers one failure mode, a camera bracket that is bumped or replaced after commissioning. The others follow from how the calibration is built, and the list is engineering reasoning from the cited procedures rather than a list any one source gives:

  • Robot kinematic error. The gripper-to-base transformation comes from the robot's kinematics, so an error in the robot's own model, a crash-damaged axis, or a wrong tool frame enters the solution as if it were camera error.
  • A pose set with parallel rotation axes. The documented minimum is 2 motions with non-parallel rotation axes; a set that fails it can return a result that looks plausible but is not unique.
  • A moved target. The procedure assumes a static pattern. A target knocked during a teach session corrupts every pose recorded after it.
  • Refocused or changed lenses. The intrinsics model the lens, so refocusing, changing the lens, or changing the working distance means the intrinsics, and then the hand-eye transform, must be redone.
  • Image and robot position out of step. For a robot-mounted camera, the part position is computed from the robot's motion, as described above. If the robot position is recorded at a different moment from the image, the result is wrong by however far the robot moved in between.

As further engineering reasoning, not a statement of the cited sources, each of these failures produces the same symptom, an offset or scatter in picks, so a recalibration should start with a check of the mechanics and the robot, not with the camera software. The procedures the failure-mode list above is drawn from are the hand-eye calibration routine's documented requirements, Shiu and Ahmad's analysis, and Zhang's camera model (OpenCV 4.13.0, calib3d, cv::calibrateHandEye Note; Shiu and Ahmad 1989, IEEE Transactions on Robotics and Automation 5-1; Zhang 2000, IEEE TPAMI 22-11).

How is a calibration verified, and when must it be repeated?​

A calibration is not proven by the solver finishing without an error. The four items below, including the recalibration triggers, are engineering practice rather than content of any cited source. A verification plan for a handling cell covers:

  • Independent check positions. Place a check target at positions that were not used for calibration, command the robot to them from the vision result, and compare against the actual target, with an acceptance limit taken from the pick or place tolerance of the application.
  • Solver comparison. Record the result of a second solution method on the same data, as described above.
  • Recalibration triggers. A bracket, lens, tool, or robot repair is a trigger to verify before production resumes. So is a crash, even a minor one.
  • Records. Keep the pose set, the images, and the result, so a later drift can be compared with the commissioning state.

This article sets no numeric acceptance limit, because the right one comes from the application's tolerance. UTEC Industrial performs factory acceptance testing and on-site commissioning, so a calibration verification step can be written into the acceptance plan and demonstrated before handover. For further reading, the Hornberg handbook's camera-calibration chapter has a section on verification of calibration results, and the robot-guidance case study in its Chapter 10 has two key-point subsections, calibration and communication; the handbook is used here at table-of-contents level only, not as the source of the list above (Hornberg 2017, Ch. 5, Verification of Calibration Results, p. 308, and Ch. 10, §10.11.5.3–§10.11.5.4, pp. 782–783).

How does the vision result move between the camera, the robot, and the PLC?​

Where the robot controller applies the vision offset to its own motion, as with the FANUC offsets described above, the PLC coordinates the cell around it: conveyors, cars, positioners, doors, and the safety system. The connection between the two follows the standard EtherNet/IP roles. ODVA defines scanner-class products, which originate I/O connections, and adapter-class products, which are the targets of those connections. ODVA lists PLCs among its scanner-class examples, and robots that send and receive real-time data at the request of PLCs and other controllers among its adapter-class examples; robots also appear among the scanner-class examples. As engineering reasoning, not an ODVA statement, the cell design therefore sets the role: where the PLC coordinates the robot, the PLC is the scanner and the robot the adapter. The article on connecting a FANUC robot to an Allen-Bradley PLC over EtherNet/IP covers the connection setup, timing, and buffering, and the category overview covers the new-data handshake.

The sensing that surrounds the camera matters as much as the camera. Apart from the event-task point, the list below is engineering practice rather than content of the cited sources:

  • Triggers. A part-present sensor or a conveyor encoder tells the camera when to image and tells the controls where the imaged part has moved to.
  • Event-driven logic. On ControlLogix 5580 and CompactLogix 5380 controllers, Rockwell Automation's controller design manual lists a consumed tag and a change in module input data among the triggers for an event task, so, as engineering reasoning, a "vision result ready" bit can start the next step without waiting for a continuous-task scan; the trigger list differs by controller family, as the category overview sets out.
  • Pick confirmation. A vacuum switch or gripper position sensor confirms the pick before the robot leaves the pick point; a vision result alone does not prove the part is held.
  • Result status. A "no part found" or a low match score has to reach the PLC as a status, so the cell can stop, retry, or divert rather than move to a stale position.

UTEC Industrial, a Rockwell Automation Recognized System Integrator, programs the Allen-Bradley ControlLogix and CompactLogix controllers and EtherNet/IP networks that tie these cells together (ODVA PUB00138R8-2024; Rockwell Automation 1756-RM094N-EN-P-2025, Ch. 5 p. 43 and Ch. 4 p. 30).

Which safety rules apply while a robot runs calibration moves?​

Calibration is the time when a robot is most likely to be moving with a person close by. The robot steps through many poses, often in a teach mode, while someone watches the target or adjusts a light. The governing standards for the cell are:

  • ISO 10218-1:2025, safety requirements for the industrial robot itself, as partly completed machinery, before it is integrated into a robot application.
  • ISO 10218-2:2025, safety requirements for industrial robot applications and robot cells, which addresses integration, commissioning, operation, maintenance, and decommissioning.
  • ANSI/A3 R15.06-2025, the American National Standard for industrial robots and robot systems, which is the US national adoption of ISO 10218 Parts 1 and 2.
  • ISO 12100:2010, which specifies principles of risk assessment and risk reduction for achieving safety in the design of machinery.
  • ISO 13849-1:2023, the design and integration of safety-related parts of control systems that perform safety functions, in high demand and continuous modes of operation; it does not apply to low demand mode.

Two points follow for vision cells. First, the calibration and verification routines are tasks the risk assessment must list, with the robot speed, the operator's position, and the enabling device defined for each; that is engineering reasoning from the standards' scope, because these routines move the robot to many poses while people are close by. Second, a production camera is not a safety device: IEC 61496-1:2020 specifies general requirements for the design, construction, and testing of non-contact electro-sensitive protective equipment designed specifically to detect persons or parts of a person as part of a safety-related system, which a part-locating camera is not. The category overview covers that limit and the lockout that cleaning a camera inside the cell requires, and the article on when a robot beats a custom mechanism covers risk assessment for heavy-handling robot cells (ISO 10218-1:2025; ISO 10218-2:2025; ANSI/A3 R15.06-2025; ISO 12100:2010; ISO 13849-1:2023; IEC 61496-1:2020).

Related Articles

References​

  • Horaud, R., & Dornaika, F. (1995). "Hand-Eye Calibration." The International Journal of Robotics Research, 14(3), 195-210. DOI 10.1177/027836499501400301
  • OpenCV 4.13.0: Camera Calibration and 3D Reconstruction (calib3d), cv::calibrateHandEye. OpenCV documentation, 2025.
  • Zhang, Z. (2000). "A flexible new technique for camera calibration." IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11), 1330-1334. DOI 10.1109/34.888718
  • FANUC B-83914EN-2/01: R-30iB Plus Controller iRVision 2D Camera Application Operator's Manual. FANUC Corporation, 2017.
  • Shiu, Y. C., & Ahmad, S. (1989). "Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form AX=XB." IEEE Transactions on Robotics and Automation, 5(1), 16-29. DOI 10.1109/70.88014
  • Park, F. C., & Martin, B. J. (1994). "Robot sensor calibration: solving AX=XB on the Euclidean group." IEEE Transactions on Robotics and Automation, 10(5), 717-721. DOI 10.1109/70.326576
  • Daniilidis, K. (1999). "Hand-Eye Calibration Using Dual Quaternions." The International Journal of Robotics Research, 18(3), 286-298. DOI 10.1177/02783649922066213
  • Tsai, R. Y., & Lenz, R. K. (1989). "A new technique for fully autonomous and efficient 3D robotics hand/eye calibration." IEEE Transactions on Robotics and Automation, 5(3), 345-358. DOI 10.1109/70.34770
  • Hornberg, A. (Ed.). Handbook of Machine and Computer Vision: The Guide for Developers and Users, 2nd ed. Wiley-VCH, 2017. ISBN 9783527413393
  • ODVA PUB00138R8-2024: EtherNet/IP — CIP on Ethernet Technology. ODVA, 2024.
  • Rockwell Automation 1756-RM094N-EN-P-2025: Logix 5000 Controllers Design Considerations. Rockwell Automation, 2025.
  • ISO 10218-1:2025: Robotics — Safety requirements — Part 1: Industrial robots. ISO, 2025.
  • ISO 10218-2:2025: Robotics — Safety requirements — Part 2: Industrial robot applications and robot cells. ISO, 2025.
  • ANSI/A3 R15.06-2025: American National Standard for Industrial Robots and Robot Systems – Safety Requirements. A3/ANSI, 2025.
  • ISO 12100:2010: Safety of machinery — General principles for design — Risk assessment and risk reduction. ISO, 2010.
  • ISO 13849-1:2023: Safety of machinery — Safety-related parts of control systems — Part 1: General principles for design. International Organization for Standardization, 2023.
  • IEC 61496-1:2020: Safety of machinery — Electro-sensitive protective equipment — Part 1: General requirements and tests. IEC, 2020 (Ed.4).

Ready to Discuss a Material Handling System?​

UTEC Industrial designs, engineers, machines, fabricates, and installs custom material handling systems for heavy industry, from the stress-relieved structure and drives to the Allen-Bradley PLC controls, tuning, and monitoring that run them, at its Spokane Valley, WA facility. Send UTEC the application, loads, and duty cycle to start a system review.

Request a Quote →

Questions? Call (509) 922-1832 or email sales@utec.co

Last updated on