Mark Humphrys - Teaching - CA425

Artificial Intelligence

---

Course Descriptor

• CA425 - Artificial Intelligence (and here)

• 70 exam - 30 practical.
• 2 hour exam

• Past exam papers

• email lists
  • ca425 mail list archive

---

How to contact me

See How to contact me.

---

Notes

My notes contain many hyperlinks to background material. Some students get confused about what is the core course. The core course is anything that is linked to directly on this front page. All other links are just background material.

1. Introduction - State-space control
   1. Introduction to AI
   2. Survey of AI
   3. AI Links
   4. Robotics Links
   5. History of AI

   6. Continuum of Autonomy
   7. State-space control
      • Sample applications of Reinforcement Learning
   8. RL as Pattern Classification

   
   

2. Reinforcement Learning - Reference

   
   

3. Reinforcement Learning
   1. Formal introduction to Reinforcement Learning (Chapter 2 of my PhD)
      1. Notation
      2. Appendix A of PhD
      3. Appendix B of PhD
      4. Appendix C - 2-reward reward functions
      5. Appendix D - 3-reward (or more) reward functions
      6. Ch.7 - Rewarding on transitions or continuously
      7. Ch.18 - Feudal Q-learning

   2. Reinforcement Learning - Accompanying Notes
      1. Exercise
      2. How Q-learning works
      3. Building a world model
      4. Convergence
      5. The control policy
      6. Boltzmann "soft max" distribution

   3. Program code
      1. How to make a decision probabilistically
      2. Coding the state-space as a lookup-table
      3. Sample code for lookup-table Q-learning

   
   

4. Movie demo
   1. Movie demo of W-learning contains within it a demo of basic Q-learning.

   
   

5. Reinforcement Learning with Neural Networks (Pre-requisite needed.)
   1. Neural Networks (Revision)
   2. Using a Neural Network as a generalisation in RL
      • Designing the Inputs
   3. Q-learning with a Neural Network
   4. Ch.4 - Using a Neural Network with RL

   
   

6. Multiple Minds
   1. Ch.3 - Multi-Module Reinforcement Learning
      • Hierarchical Q-learning
   2. Ch.4 - Multiple Minds in the same body - Test of Hierarchical Q-learning
      • Learning Q together
   3. Ch.18 - The general form of a Society of Mind based on Reinforcement Learning
   4. Open Issues in AI
   5. Architectures of Autonomous Agents
      • Reading
   6. The World-Wide-Mind (my idea)

---

Practical

Practical - Play "X's and O's" with RL

---

Reading

Experiments in Adaptive State-Space Robotics, Clocksin and Moore, 1989. - Online. - A simple introduction to the very idea of state-space robotic or agent control.

How to Make Software Agents Do the Right Thing: An Introduction to Reinforcement Learning, Singh et al, 1996. - Online. - A simple introduction to the idea of RL.

Action Selection methods using Reinforcement Learning, Humphrys, 1997 (my PhD thesis). - Online. - Chapter 2 is the more formal introduction to RL above.

Kaelbling et al (1996), "Reinforcement Learning: A Survey", Journal of Artificial Intelligence Research 4:237-285. - Online.

Reinforcement Learning: An Introduction, Sutton and Barto, 1998. - Bookshop, and Online (also here and here).

---

Library categories

• 006 - Special computer methods
  • 006.3 - Artificial Intelligence

• 519 - Probabilities & applied mathematics
  • 519.7 - Programming