Mark Humphrys - Teaching - CA425

Artificial Intelligence

---

Course Descriptor

• CA425 - Artificial Intelligence (and here)

• 70 exam - 30 practical.
• 2 hour exam
• This year: Answer 3 questions out of 4. (33 marks each. Free 1 mark.)

• Past exam papers (did not run in 2011-12 and 2010-11)
  • Exam paper 2009-10

• email lists
  • ca425 mail list archive

---

How to contact me

See How to contact me.

---

Notes

My notes contain many hyperlinks to background material. Some students get confused about what is the core course. The core course is anything that is linked to directly on this front page. All other links are just background material.

1. Background
   1. Introduction to AI
   2. Survey of AI
   3. History of AI

   4. AI Links
   5. Robotics Links

   
   

2. State-space control
   1. Continuum of Autonomy
   2. State-space control
      • Sample applications of Reinforcement Learning
   3. RL as Pattern Classification

   
   

3. Reinforcement Learning - Reference

   
   

4. Reinforcement Learning
   1. PhD extract
      1. Notation
      2. Appendix A of PhD
      3. Appendix B of PhD
      4. Ch.7 - Rewarding on transitions or continuously

   2. Accompanying Notes
      1. Exercise
      2. How Q-learning works

   3. Program code (for practical)
      1. Coding the state-space as a lookup-table
      2. Sample code for lookup-table Q-learning

   
   

5. Reinforcement Learning - More
   1. PhD extracts
      1. Appendix C - 2-reward reward functions
      2. Appendix D - 3-reward (or more) reward functions
      3. Ch.18 - Feudal Q-learning

   2. Accompanying notes
      1. Building a world model
      2. Convergence
      3. The control policy
      4. Boltzmann "soft max" distribution
      5. How to make a decision probabilistically

   
   

6. Movie demo
   1. Movie demo of W-learning contains within it a demo of basic Q-learning.

   
   

7. Reinforcement Learning with Neural Networks (Pre-requisite needed.) - NOT ON COURSE THIS YEAR

   Neural Networks (Revision) Using a Neural Network as a generalisation in RL Designing the Inputs Q-learning with a Neural Network Ch.4 - Using a Neural Network with RL

   
   

8. Multiple Minds - NOT ON COURSE THIS YEAR

   Ch.3 - Multi-Module Reinforcement Learning Hierarchical Q-learning Ch.4 - Multiple Minds in the same body - Test of Hierarchical Q-learning Learning Q together Ch.18 - The general form of a Society of Mind based on Reinforcement Learning Open Issues in AI Architectures of Autonomous Agents Reading The World-Wide-Mind (my research project)

---

Notes on Assignment Notation

I often use   :=   for assignment to distinguish from   =   for equality.

Notes on Assignment Notation

---

Labs

If the practical is based on the WWM server, I will hold one or two hands-on labs. Dates will be announced.

---

Practical

Practical - Play "X's and O's" with RL

Deadline Wed 12 Dec 2012.

---

Reading

Experiments in Adaptive State-Space Robotics, Clocksin and Moore, 1989. - Online. - A simple introduction to the very idea of state-space robotic or agent control.

How to Make Software Agents Do the Right Thing: An Introduction to Reinforcement Learning, Singh et al, 1996. - Online. - A simple introduction to the idea of RL.

Action Selection methods using Reinforcement Learning, Humphrys, 1997 (my PhD thesis). - Online. - Chapter 2 is the more formal introduction to RL above.

Kaelbling et al (1996), "Reinforcement Learning: A Survey", Journal of Artificial Intelligence Research 4:237-285. - Online.

Reinforcement Learning: An Introduction, Sutton and Barto, 1998. - Bookshop, and Online (also here and here).

---

Library categories

• 006 - Special computer methods
  • 006.3 - Artificial Intelligence

• 519 - Probabilities & applied mathematics
  • 519.7 - Programming