Skip to main content
Intended for healthcare professionals
Restricted access
Research article
First published online January 1, 2008

Ultrasound Image-Based Visual Servoing of a Surgical Instrument Through Nonlinear Model Predictive Control

Mickaël Sauvée, Philippe Poignet, and Etienne DombreView all authors and affiliations
Volume 27, Issue 1
https://doi.org/10.1177/0278364907082269

Abstract

Ultrasound image-guided interventions are widespread in surgery because of the non-invasive character of the procedures. However, hand/eye synchronization is relatively difficult for a surgeon. Ultrasound image-based visual servoing is one way to perform this kind of surgery. In this work, the control of instrument motion based on ultrasound images through nonlinear model predictive control is investigated. This new scheme ensures the convergence of the instrument to the desired position and also offers the possibility of satisfying constraints such as joint limits, actuator saturation and visibility preserving. This paper describes the proposed controller. The efficiency and the robustness of the proposed solution to control a six degree-of-freedom mechanical system is first illustrated by simulation. Experiments on a Mitsubishi PA10 robot highlight the efficiency of the vision control scheme to handle constraints of ultrasound image-based visual servoing.

Get full access to this article

View all access and purchase options for this article.

Get Access

References

Abolmaesumi, P., Salcudean, S., Zhu, W., Sirouspour, M., and DiMaio, S. (2001). Imageguided control of a robot for medical ultrasound . IEEE Transactions on Robotics and Automation, 18(1): 11—23.
Google Scholar
Allgöwer, F., Badgwell, T.A., Qin, S.J., Rawlings, J.B., and Wright, S.J. (1999). Nonlinear predictive control and moving horizon estimation — an introductory overwiew. In Advances in Control: Highlights of ECC'99, pp. 391—449.
Google Scholar
Batcha, W. and Krupa, A. (2006). Towards ultrasound image-based visual servoing . In Proceedings of IEEE International Conference on Robotics and Automation (ICRA'06), pp. 4112—4117.
Google Scholar
Chavanon, O., Carrat, L., Pasqualini, C., Dubois, E., Blin, D., and Troccaz, J. (2000). Computer-guided pericardiocentesis: experimental results and clinical perspectives. Herz, 25(8): 761—768.
Google Scholar
Diegeler, A., Hirsch, R., Schneider, F., Schilling, L., Falk, V., Rauch, T., and Mohr, F. (2000). Neuromonitoring and neurocognitive outcome in off-pump versus conventional coronary bypass operation. Annals of Thoracic Surgery, 69: 1162—1166.
Google Scholar
Downing, S.D., Herzog, W.A., McLaughlin, J.S., and Gilbert, T.P. (2002). Beating-heart mitral valve surgery: preliminary model and methodology. Journal of Thoracic and Cardiovascular Surgery, 123(6): 1141—1146.
Google Scholar
Hastenteufel, M., Yang, S., Christoph, C., Vetter, M., Meinzer, H.-P., and Wolf, I. (2006). Image-based guidance for minimally invasive surgical fibrillation ablation. International Journal of Medical Robotics and Computer Assisted Surgery, 2: 60—69.
Google Scholar
Hong, J., Dohi, T., Hashizume, M., Konishi, K., and Hata, N. (2004). An ultrasound-driven needle insertion robot for percutaneous cholecystostomy. Physics in Medicine and Biology, 49(3): 441—455.
Google Scholar
Hutchinson, S., Hager, G.D., and Corke, P. (1996). A tutorial on visual servo control. IEEE Transactions on Robotics and Automation, 12(5): 651—670.
Google Scholar
Lawrence, G., Zhou, J.L., and Tits, A.L. (1997). User's guide for CFSQP (Version 2.5).
Google Scholar
Megali, G., Tonet, O., Stefanini, C., Boccadoro, M., Papaspyropoulos, V., Angelini, L., and Dario, P. (2001). A computer-assisted robotic ultrasound-guided biopsy system for videoassisted surgery. In Proceedings of Medical Imaging Computing and Computer-Assisted Intervention (MICCAI'01) .
Google Scholar
Phee, L., Xiao, D., Yuen, J., Chan, C.F., Ho, H., Thng, C.H., Cheng, C., and Ng, W.S. (2005). Ultrasound guided robotic system for transperineal biopsy of the prostate. In Proceedings of IEEE International Conference on Robotics and Automation (ICRA'05), pp. 1327—1332.
Google Scholar
Picone, A.L., Lutz, C.J., Finck, C., Carney, D., Gatto, L.A., Paskanik, A., Searles, B., Snyder, K., and Nieman, G. (1999). Multiple sequential insults cause post-pump syndrome. Annals of Thoracic Surgery, 67: 978— 985.
Google Scholar
Rastan, A.J., Bittner, H.B., Gummert, J.F., Walther, T., Schewick, C.V., Girdauskas, E., and Mohr, F.W. (2005). On-pump beating heart versus off-pump coronary artery bypass surgery — evidence of pump-induced myocardial injury. European Journal of Cardio-Thoracic Surgery, 27: 1057—1064.
Google Scholar
Richalet, J. (1993). Pratique de la commande prédictive, Chapter 4, pp. 193—199. Hermès.
Google Scholar
Sauvée, M., Poignet, P., and Dombre, E. (2006). Towards nonlinear model predictive control for image based visual servoing. In Proceedings of IFAC Workshop on Nonlinear Model Predictive Control for Fast System (NMPCFS'06), pp. 147—152.
Google Scholar
Schneider, O., Troccaz, J., Chavanon, O., and Blin, D. (2000). PADyC: a synergistic robot for cardiac puncturing . In Proceedings of IEEE International Conference on Robotics and Automation (ICRA'00), Volume 3, San Fransisco, USA, pp. 2883—2888.
Google Scholar
Stoll, J., Novotny, P., Howe, R., and Dupont, P. (2006). Real-time 3d ultrasound-based servoing of a surgical instrument. In Proc. IEEE Int. Conf. on Robotics and Automation (ICRA'06), pp. 613—618.
Google Scholar
Vitrani, M.-A., Morel, G., and Ortmaier, T. (2005). Automatic guidance of a surgical instrument with ultrasound based visual servoing. In Proceedings of IEEE International Conference on Robotics and Automation (ICRA'05), pp. 508— 513.
Google Scholar
Wan, I.Y.P., Arifi, A.A., Wan, S., Yip, J.H.Y., Sihoe, A.D.L., Thung, K.H., Wong, E.M.C., and Yim, A.P.C. (2004). Beating heart revascularization with or without cardiopulmonary bypass: evaluation of inflammatory response in a prospective randomiezd study. Journal of Thoracic and Cardiovascular Surgery, 127(6): 1624—1631.
Google Scholar

Cite article

Cite article

Cite article

OR

Download to reference manager

If you have citation software installed, you can download article citation data to the citation manager of your choice

Share options

Share

Share this article

Share with email
EMAIL ARTICLE LINK
Share on social media

Share access to this article

Sharing links are not relevant where the article is open access and not available if you do not have a subscription.

For more information view the Sage Journals article sharing page.

Information, rights and permissions

Information

Published In

Volume 27, Issue 1
Pages: 25 - 40
Article first published online: January 1, 2008
Issue published: January 2008

Keywords

  1. human-centered and life-like robotics
  2. medical robots and systems
  3. mechanics
  4. design and control
  5. motion control
  6. sensing and perception
  7. computer vision
  8. visual tracking

Rights and permissions

Request permissions for this article.

Authors

Affiliations

Mickaël Sauvée
LIRMM (Laboratoire d'informatique, de Robotique et de Microélectronique de Montpellier), UMR 5506 CNRS Université de Montpellier 2 161 rue Ada, 34392 Montpellier Cedex 5, France, sauvee, @lirmm.fr
View all articles by this author
Philippe Poignet
LIRMM (Laboratoire d'informatique, de Robotique et de Microélectronique de Montpellier) UMR 5506 CNRS Université de Montpellier 2 161 rue Ada, 34392 Montpellier Cedex 5, France, poignet, @lirmm.fr
View all articles by this author
Etienne Dombre
LIRMM (Laboratoire d'informatique, de Robotique et de Microélectronique de Montpellier) UMR 5506 CNRS Université de Montpellier 2 161 rue Ada, 34392 Montpellier Cedex 5, France, [email protected]
View all articles by this author

Metrics and citations

Metrics

Journals metrics

This article was published in The International Journal of Robotics Research.

VIEW ALL JOURNAL METRICS

Article usage*

Total views and downloads: 142

*Article usage tracking started in December 2016

Altmetric

See the impact this article is making through the number of times it’s been read, and the Altmetric Score.
Learn more about the Altmetric Scores


Articles citing this one

Receive email alerts when this article is cited

Web of Science: 33 view articles Opens in new tab

Crossref: 39

  1. Online Predictive Visual Servo Control for Constrained Target Tracking...
    Go to citation Crossref Google Scholar
  2. An Uncalibrated Image-Based Visual Servo Strategy for Robust Navigatio...
    Go to citation Crossref Google Scholar
  3. A Visual Predictive Control Framework for Robust and Constrained Multi...
    Go to citation Crossref Google Scholar
  4. Adaptive Image-based Visual Servoing with Time-varying Learning Rates ...
    Go to citation Crossref Google Scholar
  5. Conjugated Visual Predictive Control for Constrained Visual Servoing
    Go to citation Crossref Google Scholar
  6. Disturbance observer based adaptive model predictive control for uncal...
    Go to citation Crossref Google Scholar
  7. Depth-based Visual Predictive Control of Tendon-Driven Continuum Robot...
    Go to citation Crossref Google Scholar
  8. Intelligent magnetic manipulation for gastrointestinal ultrasound
    Go to citation Crossref Google Scholar
  9. Model Predictive Control for Constrained Image‐Based Visual Servoing i...
    Go to citation Crossref Google Scholar
  10. Model Predictive Control for Uncalibrated and Constrained Image-Based ...
    Go to citation Crossref Google Scholar
  11. Photoacoustic-based visual servoing of a needle tip
    Go to citation Crossref Google Scholar
  12. Photoacoustic based visual servoing of needle tips to improve biopsy o...
    Go to citation Crossref Google Scholar
  13. Moments-Based Ultrasound Visual Servoing: From a Mono- to Multiplane A...
    Go to citation Crossref Google Scholar
  14. Cooperative Control with Ultrasound Guidance for Radiation Therapy
    Go to citation Crossref Google Scholar
  15. A robust self triggered Image Based Visual Servoing Model Predictive C...
    Go to citation Crossref Google Scholar
  16. Visual Predictive Control of Spiral Motion
    Go to citation Crossref Google Scholar
  17. Predictive control architecture for real-time image moments based serv...
    Go to citation Crossref Google Scholar
  18. Design and implementation of visual predictive control for image posit...
    Go to citation Crossref Google Scholar
  19. Robustness analysis of model predictive control for constrained Image-...
    Go to citation Crossref Google Scholar
  20. Guidance of a high dexterity robot under 3D ultrasound for minimally i...
    Go to citation Crossref Google Scholar
  21. Bibliography
    Go to citation Crossref Google Scholar
  22. Stabilizing Predictive Visual Feedback Control via Image Space Navigat...
    Go to citation Crossref Google Scholar
  23. Intensity-Based Ultrasound Visual Servoing: Modeling and Validation Wi...
    Go to citation Crossref Google Scholar
  24. Developments in Visual Servoing for Mobile Manipulation
    Go to citation Crossref Google Scholar
  25. Robotic ultrasound systems in medicine
    Go to citation Crossref Google Scholar
  26. Vision‐based Control
    Go to citation Crossref Google Scholar
  27. Predictive control of nonlinear visual servoing systems using image mo...
    Go to citation Crossref Google Scholar
  28. A system for 3D ultrasound-guided robotic retrieval of foreign bodies ...
    Go to citation Crossref Google Scholar
  29. Predictive control architecture for real-time image moments based serv...
    Go to citation Crossref Google Scholar
  30. Evolution of autonomous and semi‐autonomous robotic surgical systems: ...
    Go to citation Crossref Google Scholar
  31. Increasing the robustness of Acrobot walking control using compliant m...
    Go to citation Crossref Google Scholar
  32. Collision-free visual servoing of an eye-in-hand manipulator via const...
    Go to citation Crossref Google Scholar
  33. Intensity-based direct visual servoing of an ultrasound probe
    Go to citation Crossref Google Scholar
  34. A multi-plane approach for ultrasound visual servoing: Application to ...
    Go to citation Crossref Google Scholar
  35. An autonomous mobile grasping system using visual servoing and nonline...
    Go to citation Crossref Google Scholar
  36. Mechanical Design of a Needle Driven Robot
    Go to citation Crossref Google Scholar
  37. 2-D Ultrasound Probe Complete Guidance by Visual Servoing Using Image ...
    Go to citation Crossref Google Scholar
  38. The robot system of radio frequency ablation based on ultrasound navig...
    Go to citation Crossref Google Scholar
  39. Modeling and 3D local estimation for in-plane and out-of-plane motion ...
    Go to citation Crossref Google Scholar

Figures and tables

Figures & Media

Tables

View Options

Get access

Access options

If you have access to journal content via a personal subscription, university, library, employer or society, select from the options below:

IOM3 members can access this journal content using society membership credentials.

MEMBERSHIP LOGIN

IOM3 members can access this journal content using society membership credentials.

MEMBERSHIP LOGIN

Alternatively, view purchase options below:

Purchase 24 hour online access to view and download content.

Subscribe to this journal

Access journal content via a DeepDyve subscription or find out more about this option.

Start 2 week free trial
Need help?

View options

PDF/ePub

View PDF/ePub
Download PDF

Also from Sage