Dana H. Ballard

PhD, University of California, Irvine, 1974

Professor, Computer Science

Department of Computer Sciences
University of Texas at Austin
Austin TX 78712

(512) 471 9750

contact

Office Hours

By appointment.

Back to Contents

My main research interest is in computational theories of the brain with emphasis on human vision and motor control. In 1985 Chris Brown and I led a team that designed and built a high speed binocular camera control system capable of simulating human eye movements. The system was mounted on a robotic arm that allowed it to move at one meter per second in a two meter radius workspace. This system has led to an increased understanding of the role of behavior in vision, in particular that visual computations can be simpler when interacting in the 3D world. To pursue this avenue we became interested in pursuing this research by using high DOF models of humans' natural behavior in virtual reality environments. To be able to monitor human eye movements in these enviroments, with Mary Hayhoe and Jeff Pelz we build the first head montor that ecorporated a eye tracker inside the HMD. My PhD research focused on biomedical image processing but as an assistant professor I became more interested on the general problem of modeling vision and collaborated with my University of Rochester colleague Chris Brown to write the first text in computer vision in 1982, which is still accessible on the web. In the middle 80s the field discovered biologically motivated active vision could be much more efficient than static picture analysis. Chris and I built the first real-time binocular robot that could make saccades and pursue moving targets. The robot was enormously influential in showing that motion actually made many of the vision computations simpler by providing efference copy information. As a result, my interests progressed to the human brain realized vision. Through Rochester's Bridging program to learn tract tracing in visual cortex in cats and rats in the department of Anatomy and Physiology. I became hooked on the brain, but realized my talents were in mathematical modelingThat experience has led to a lifelong interest in neural mode . A brilliant student, Raj Rao was able to define a predictive coding model that showed that neural systems spanning cortical maps could be seen as pivotal in learning memory representations 1. That model has been enormously successful, but a frustration for me has been that it was cast at a level above spikes. Working with a postdoc, Janneke Jehee, we put together a spiking model, which exhibited several desirable properties, but had unsolved problems. Most recently I have resolved these problems with another star student Ruohan Zhang. We realized that we are at the stage that the elements of the spiking model could be tested. In that process, I have been extraordinarily fortunate in starting a collaboration with Luc Genet who is a leader in cell patch clamp technology, particularly in the two-cell patches we will need for next step. We have been collaborating for the two years and our joint efforts are extremely promising. This effort dovetails with a second interest in How the brain represents movement. Our newest papers show that humans use common postures in whole body tasks and that these are local energy minima, suggesting new constraints for motor cortex. 1. Friston, K., Nature Neuroscience 2018 1019-1026
Back to Contents

• Computer Science CS391L MACHINE LEARNING
  
  
• Computer Science 342C COMPUTATIONAL BRAIN

Back to Contents

• Ballard, D.H., G.E. Hinton, and T.J. Sejnowski (1983), Parallel visual computation, Nature 306, 5938, 21-26, 3
• Ballard, D., Hayhoe, M., Pook, P., & Rao, R. (1997). Deictic codes for the embodiment of cognition. Behavioral and Brain Sciences, 20, 723-767. [PDF]
• Ballard, D., Hayhoe, M., & Pelz, J. (1995). Memory representations in natural tasks. Cognitive Neuroscience, 7, 66-80.
• Ballard, D.H., "Animate vision," Artificial Intelligence Journal 48, 57-86, 1991
• Tatler, B. W., Hayhoe, M.M., Land, M. F., and Ballard, D. H. (2011) Eye Guidance in natural vision: Reinterpreing salience, ournal of Vision[PDF]



Active Vision

Research directed towards agendra-driven models of visual routines.
• Hayhoe, M. M., and Ballard, D. H. (2005) Eye Movements in Natural Behavior, Trends in Cognitive Science [PDF]
• Ballard, D.H., "Generalizing the Hough transform to detect arbitrary shapes," Pattern Recognition 13, 2, April 1981[PDF
• Swain, M. J. & Ballard, D. H. (1991) Color Indexing, International Journal of Computer Vision, 7,1,11-32 [PDF]
• Yi, W. and Ballard, D. H. (2009) Recognizing behavior in hand-eye coordination patterns, Int. Journal of Humanoid Robotics[PDF]
• Ballard, D. H. and Hayhoe, M. M.(2009) Modeling the role of task in the control of gaze, Visual Cognition, 17, 1185-1204
• Rothkopf, C. A., and Ballard, D. H.(2009) Image statistics at the point of gaze during human navigation, Visual Neuroscience,26, 81-92
• Rothkopf, C. A., Ballard D. H. and Hayhoe, M. M. (2007) Task and context determine where you look, Journal of Vision, 7(14) 1-20 [PDF]
Triesch, J., Ballard, D., Hayhoe, M., & Sullivan, B. (2003). What you see s what you need. Journal of Vision, 3, 86-94.[PDF]


Neural Coding

Spike based models of neuron function. Spikes are modulationed by the gamma frequency. Newest modules use thelocal gamma phase
• Rao, R.P.N. and D.H. Ballard,(1999), Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects, Nature Neuroscience 2, 1, 79 [PDF]
• Ballard, D. H., and Jehee, J. M. F. (2012)Dynamic coding of signed quantities in cortical feedback circuits, Frontiers in Perception Science[PDF]
• Ballard, D. H., and Jehee, J. M. F. (2011) Dual roles for spike signaling in cortical neural populations, Frontiers in Computational Neuroscience[PDF]
• Jehee, J. F. M. and Ballard, D. H.(2009) Predictive Feedback Can Account for Biphasic Responses in the Lateral Geniculate Nucleus, PLoS Computational Biology[PDF]
• Ruohan Zhang, Dana H Ballard (2020) Parallel neural multiprocessing with gamma frequency latencies, Neural Computation The first genetative of gamma phase scalar representations of spikes.
  

  Reinforcement Learning with modules

  Reinforcement learning for large state spaces can be prehensive expensitive. A factoring of the state space into linear cobinations turns out to have several important consequences. B. T. Sullivan, L. Johnson, C. A. Rothkopf, D. H. Ballard and M. M. Hayhoe, (2012)The role of uncertainty and reward on eye movements in a virtual driving task, Journal of Vision,12(13):19, 1-16[PDF]
• Rothkopf, C. A., and Ballard, D. H. (2010) Credit assignment in multiple goal embodied visuomotor behavior, Frontiers in Psychology[PDF]
A new module co-active with trained modules can use theirr information to train it.
• C. A. Rothkopf and Ballard, D. H.(2013)Modular inverse reinforcement learning for visuomotor behavior, Biological Cybernetics, 107(4),477-490[PDF]
• Ballard, D. H., Kit, D., Rothkopf, C. A. and Sullivan, B. (2013)A hierarchical modular architecture for embodied cognition, Multisensory Research,26(1-2),177-204[PDF]
• Ruohan Zhang, Shun Zhang, Matthew H Tong, Yuchen Cui, Constantin A Rothkopf, Dana H Ballard, Mary M Hayhoe(2013)Modeling sensory-motor decisions in natural behavior, PLoS Computational Biology,26(1-2),177-204[PDF]
Sprague, N. and Ballard, D. H. (2018) Modeling Embodied Visual Behaviors ACM Transactions on Applied Perception Sprque's insight shows that modules can compete for gaze on the basis of the variances of their reforcements rather than reinforcement.
• Leif Johnson, Brian Sullivan, Mary Hayhoe and Dana Ballard,(2014)Predicting human visuomotor behaviour in a driving task,Phil. Trans. R. Soc.,369,20130044.[PDF
Johson uses eye tracing in a virtual reality venue to show that Sprague's model near exactly fits human data


Neural models of Motor Control

Whole body modlies allow joint torques to be recovered. These in turn allow representations for M1 candidates to be studied as minum trajectories are a vstially small subset of all trajectories..
• Lijia Liu, Leif Johnson, Oran Xohar, and Ballard, D. H.(20219)Humans use minimum cost movements in a whole-body tasHumans Use Similar Posture Sequences in a Whole-Body Tracing Taskk, IScience, [ This result is very signicant as it suggests that movements can be precomputed and stored.
  
• Lijia Liu and Ballard, D. H.(2021)Humans use minimum cost movements in a whole-body task, Scientific Reports, [PDF]
This result is very signicant as it suggests that movements can be precomputed and stored.

• Lijia Li, Cooper, J.C. and Ballard, D. H.(2021)Computational Modeling: Human Dynamic Model, , [PDF]
• Gu, X., and Ballard, D. H. (2006) An Equilibrium Point based Model Unifying Movement Control in Humanoids, Robotics: Science and Systems [PDF]

Genetic Programing

Whole body modlies allow joint torques to be recovered. These in turn allow representations for M1 candidates to be studied.Rosca used the value of leaves fo the Lisp tree to turn them into macros that functioned as subroutines. The result was the enormpous improvement in performance.
• Rosca, J.P and Ballard, D.H.(1994) Hierarchical self-organization in genetic programming[PDF]
• Rosca, J.P (1995) Causality in Genetic Programming. IGCA
  

  models of Early word learning

  Yu changed word learning focus from statistical audio correlations to a shared attention between the child and caregiver. He and linda Smith have fkeshed out thiis paradigm
• Yu, Chen and Dana H. Ballard,(2004), A Multimodal Learning Interface for Grounding Spoken Language in Sensorimotor Experience, ACM Transactions on Applied Perception. 1, 1[PDF]
• Yu, Chen and Dana H. Ballard,(2004), A Multimodal Learning Interface for Grounding Spoken Language in Sensorimotor Experience, ACM Transactions on Applied Perception. 1, 1[PDF]
• 
  

  General

  .
• Shimozaki, S., Zelinsky, G., Hayhoe, H., Merigan, W., & Ballard, D. (2003). Spatial memory and saccade targeting deficits from parietal injury. Neuropsychologia, 41, pp. 1365-1386.
• Rao, R., Zelinsky, G., Hayhoe, M., & Ballard, D. (2002). Eye movements in iconic visual search. Vision Research, 42(11), 1447-1463. [PDF]
• Triesch, J., Ballard, D.H., and Jacobs, R.A. (2002) Fast temporal dynamics of visual cue integration. Perception, 31, 421-434 [PDF]
• Triesch, J., Sullivan, B., Hayhoe, M., & Ballard, D. (2002). Saccade contingent scene changes in unconstrained virtual reality. Proceedings, Eye Tracking Research & Application, 95-102. [PDF]
• Hayhoe, M., Bensinger, D., & Ballard, D. (1998). Task constraints in visual working memory. Vision Research, 38, 125-137. [PDF]
• de Sa, V.R., & Ballard, D. (1998) Category Learning through Multi-Modality Sensing, Neural Computation 10(5)[PDF]
A nice algorithm for co-linearing correlated audio and visual feartures
• Zelinsky, G., Rao, R., Hayhoe, M., & Ballard, D. (1997). Eye movements reveal the spatio-temporal dynamics of visual search. Psychological Science, 8, 448-453.[PDF]
• Rao, R., Zelinsky, G., Hayhoe, M., & Ballard, D. (1996). Modelling saccade targeting in visual search. In D. Touretzky, M. Mozer, & M. Hasselmo (Eds). Advances in Neural Information Processing Systems, 8, pp. 830-836. Cambridge, MA: MIT Press.
• Smeets, J., Hayhoe, M., & Ballard, D. (1996). Influence of hand movements on eye-head coordination. Experimental Brain Research, 109, 434-440.
• Ballard, D. H. (1981), Strip trees: a hierarchical representation for curves, Communications of the ACM, v.24 n.5,310-321[PDF]
A quite alogthm. ]

Ballard D. H. Brain Computation as Hierarchical Abstraction, MIT Press, 2015

Computer Vision Ballard D. H. and Brown C. M., Computer Vision, Prentice Hall, 1982

Ballard D. H. Natural Computation, MIT Press, 1987

Back to Contents

Mary Hayhoe, Professor, Department of Psychology, University of Texas at Austin
Chen Yu, Professor, Department of Psychology,University of Texas at Austin

Back to Contents

This research has received support from the National Institutes of Health and the National Sciences Foundation.

Back to Contents

Lijia Liu
Ruohan Zhang
Back to Contents