Mark B. Ring, Ph.D.

Co-founder and CEO of Cogitai, Inc., an AI startup focused on continual learning. I spend any remaining time with my wife Amy and nine-year-old son.

Last update: May 17, 2016

Research Interests:

My research revolves around a single focus: Continual Learning in Artificial Intelligence, which tries to answer one queston: If you can give an agent a single algorithm at its inception and then stand back and let it learn forever after that all on its own, what do you put into that algorithm to allow the agent to continue to learn, develop and improve forever? How should an artificial agent begin the unending process of learning and development, so that it is constantly improving its ability to comprehend and interract with the world? My 1994 dissertation, Continual Learning in Reinforcement Environments, explored this and related issues in depth. Although many ideas discussed in the dissertation have more recently fallen into favor, at the time of their publication, much of the work was far from the beaten path.

There are many potential mechanisms for artificial continual learning, but the first one I developed was called the Temporal Transition Hierarchies (TTH) algorithm, which was the first temporal function approximator that intelligently and incrementally increased history length to resolve contradictions. TTH's form expectation hierarchies, and as such represent a (perhaps unwitting) predecessor to predictive state representations (PSRs) in that they explicitly encode all future action and observation trajectories as contingencies over intervening actions and observations. The algorithm, which was also presented as an incremental method for learning FSAs (NIPS 5, 1992), was first and foremost intended for use in reinforcement environments (SAB 2, 1992).

More recent work has focused on organizing behaviors according to their similarities (using the "Motmap") and making predictions about long-term behavior-dependent predictions (Forecasts).

I've also done some very interesting work with Laurent Orseau on the safety of infinitely powerful AI (and not-quite-so-powerful AI), and explored some potentially deep philosophical issues that can, for the first time, be studied formally using methods based on the theory of computation.

Papers:

2013
: [pdf] Tom Schaul, Mark Ring. Better Generalization with Forecasts. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), Beijing, China, 2013. Abstract | Bibtex | pdf; [pdf] Mark Ring, Tom Schaul. Organizing Behavior into Spatial and Temporal Neighborhoods. This is a revised and corrected version of the 2012 ICDL paper (below), presented to the AAAI Spring Symposium on Lifelong Learning, 2013. Abstract | Bibtex | pdf; 2012
: [pdf] Mark Ring, Tom Schaul. The Organization of Behavior into Spatial and Temporal Neighborhoods. In Proc. Joint IEEE International Conference on Development and Learning (ICDL) and on Epigenetic Robotics (ICDL-EpiRob 2012), San Diego, CA, 2012. (Please see the above 2013 revised version of this paper.) Abstract | Bibtex | pdf; [pdf] Linus Gisslen, Mark Ring, Matthew Luciw, Jürgen Schmidhuber. Modular Value Iteration through Regional Decomposition. AGI, 2012. Abstract | Bibtex | pdf; [pdf] Laurent Orseau, Mark Ring. Space-Time Embedded Intelligence. AGI, 2012. Abstract | Bibtex | pdf; [pdf] Laurent Orseau, Mark Ring. Memory Issues of Intelligent Agents. AGI, 2012. Abstract | Bibtex | pdf; 2011
: [pdf] Mark Ring, Tom Schaul, Jürgen Schmidhuber. The Two-Dimensional Organization of Behavior. In Proc. Joint IEEE International Conference on Development and Learning (ICDL) and on Epigenetic Robotics (ICDL-EpiRob 2011), Frankfurt, 2011. Abstract | Bibtex | pdf; [no pdf] Hung Ngo, Mark Ring, Jürgen Schmidhuber. Curiosity Drive based on Compression Progress for Learning Environment Regularities. In Proc. Joint IEEE International Conference on Development and Learning (ICDL) and on Epignetic Robotics (ICDL-EpiRob 2011), Frankfurt, 2011. online; [pdf] Matt Luciw, Vincent Graziano, Mark Ring, Jürgen Schmidhuber. Artificial Curiosity with Planning for Autonomous Visual and Perceptual Development. In Proc. Joint IEEE International Conference on Development and Learning (ICDL) and on Epigenetic Robotics (ICDL-EpiRob 2011), Frankfurt, 2011. Abstract | Bibtex | pdf; [pdf] Mark Ring. Recurrent Transition Hierarchies for Continual Learning: A general overview. AAAI Workshops
at the Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011. Abstract | Bibtex | pdf; [pdf] Laurent Orseau, Mark Ring. Self-Modification and Mortality in Artificial Agents. AGI, 2011. Abstract | Bibtex | pdf; [pdf] Mark Ring, Laurent Orseau. Delusion, Survival, and Intelligent Agents. AGI, 2011. Abstract | Bibtex | pdf; [pdf] Leo Pape, Faustino Gomez, Mark Ring, Jürgen Schmidhuber. Modular deep belief networks that do not forget. International Joint Conference on Neural Networks (IJCNN-2011, San Francisco), 2011. Abstract | Bibtex | pdf; [pdf] Mark Ring, Tom Schaul. Q-error as a Selection Mechanism in Modular Reinforcement-Learning Systems. Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI-2011, Barcelona), 2011. Abstract | Bibtex | pdf; [pdf] Sun Yi, Faustino Gomez, Mark Ring, Jürgen Schmidhuber. Incremental Basis Construction from Temporal Difference Error. Proceedings of the 28th International Conference on Machine Learning (ICML-11), 2011. Abstract | Bibtex | pdf

2005

[pdf] Eddie J. Rafols, Mark B. Ring, Richard S. Sutton, Brian Tanner. Using Predictive Representations to Improve Generalization in Reinforcement Learning, Proceedings of the 19^th International Joint Conference on Artificial Intelligence, 2005. Abstract | Bibtex

[pdf] Mark B. Ring. Toward a formal framework for Continual Learning, post-NIPS workshop on Inductive Transfer, 2005.

1997; [pdf] Mark Ring. RCC Cannot Compute Certain FSA, Even with Arbitrary Transfer Functions, from Advances in Neural Information Processing Systems 10 (NIPS 10), 1997. Abstract | Bibtex
[pdf] [ps] [ps.Z]
: [pdf] Mark Ring. CHILD: A First Step Towards Continual Learning, Machine Learning Journal, vol. 28, 1997. Also appears as Chapter 11 in Learning to Learn, S. Thrun and L. Pratt, editors. Abstract | Bibtex
[pdf] [ps] [ps.gz]; 1996; [pdf] Mark Ring. Finding Promising Exploration Regions by Weighting Expected Navigation Costs, GMD Technical Report, Arbeitspapiere der GMD 987, April, 1996. Abstract | Bibtex
[pdf] [ps] [ps.Z]; 1995; [pdf] Mark Ring. Finding promising exploration regions by weighting expected navigation costs, working notes for talk given at the AAAI symposium on Active Learning, 1995. Abstract
[pdf] [ps] [ps.Z]; 1994; [pdf] Mark Ring. Continual Learning in Reinforcement Environments, University of Texas at Austin dissertation, 1994. See my dissertation page for more information. Abstract | Bibtex
[pdf]; 1993; [pdf] Mark Ring. Sequence Learning with Incremental Higher-Order Neural Networks, University of Texas at Austin AI lab technical report, 1993. Abstract | Bibtex
[pdf] [ps] [ps.Z]; 1992; [pdf] Mark Ring. Learning Sequential Tasks by Incrementally Adding Higher Orders, from Advances in Neural Information Processing Systems 5 (NIPS5), 1993. Abstract | Bibtex
[pdf] [ps] [ps.Z]

[pdf] Mark Ring. Two Methods for Hierarchy Learning in Reinforcement Environments, in From Animals to Animats 2: Proceedings of the Second International Conference on Simulation of Adaptive Behavior (SAB '92), proceedings dated 1993. Abstract | Bibtex

[pdf] [ps] [ps.Z]

1991

Mark Ring. Incremental Development of Complex Behaviors through Automatic Construction of Sensory-motor Hierarchies, from the proceedings of the Eighth International Workshop (ML91), 1991. Abstract | Bibtex

[ps] [ps.Z]

Contact Information

Mark B. Ring, Ph.D.

Email: "info" (use host: markring.com)

Mark B. Ring