**Note: The test on December 8 at 3pm will be held in BA B024, not the usual
lecture/tutorial room.**

**Instructor:**

Radford Neal,Office:SS6016A,Phone:(416) 978-4970,Email:radford@cs.utoronto.ca,Office hours:Tuesdays 2:10-3:00.

**Lectures:**

Mondays and Wednesdays, 3:10pm to 4:00pm, from September 11 to December 6, except for Thanksgiving (October 9). Held in BA1220.

**Tutorials:**

Fridays, 3:10pm to 4:00pm, from September 22 to December 8. Held in BA1220.

TA:Renqiang Min

NOTE:Tests will be held during the tutorial time, so youmust nothave a time conflict with the tutorial.

**Evaluation:**

Three in-class tests, worth 20% each.Tentativelyscheduled for October 13, November 17, and December 8.Four assignments, worth 10% each.

Tentativelydue on October 11, November 1, November 15, and November 29 (handed out two weeks earlier).

**Textbook:**

There is no textbook for this course. Lecture slides will be available from this web page. I will also post links to various on-line references.

**Computing**:

The assignments will involve writing programs in Matlab/Octave or R. You can download Octave and R for your home computer for free from their web sites. You can also obtain an account on the CS department's CDF computer system, where R and Matlab are available - just type "R" or "matlab" (or "matlab -nosplash -nojvm" for a less GUI version).

**Lecture slides:**

The slides are available in Postscript (ps) and Portable Document Format (pdf), in both one-per-page and four-per-page (4up) versions.Introduction: ps, pdf, ps-4up, pdf-4up

Nearest-neighbor and linear regression: ps, pdf, ps-4up, pdf-4up

Probability and loss functions: ps, pdf, ps-4up, pdf-4up

Naive Bayes classifiers: ps, pdf, ps-4up, pdf-4up

Logistic regression (revised and extended): ps, pdf, ps-4up, pdf-4up

Decision trees (revised and extended): ps, pdf, ps-4up, pdf-4up

Cross validation: ps, pdf, ps-4up, pdf-4up

Neural networks: ps, pdf, ps-4up, pdf-4up

Maximum likelihood estimation for neural networks: ps, pdf, ps-4up, pdf-4up

Training neural networks with early stopping: ps, pdf, ps-4up, pdf-4up

Bayesian learning: ps, pdf, ps-4up, pdf-4up

Bayesian neural networks: ps, pdf, ps-4up, pdf-4up

Markov chain Monte Carlo: ps, pdf, ps-4up, pdf-4up

Clustering: ps, pdf, ps-4up, pdf-4up

Mixture models (revised and extended): ps, pdf, ps-4up, pdf-4up

Principal Component Analysis (revised): ps, pdf, ps-4up, pdf-4up

Factor Analysis (extended): ps, pdf, ps-4up, pdf-4up

Nonlinear dimensionality reduction: ps, pdf, ps-4up, pdf-4up

**Assignments**

Assignment 1:Handout: ps, pdf.Assignment 2:

Due date has been extended to October 18.

Data set 1: train x, train y, test x, test y.

Data set 2: train x, train y, test x, test y.

Maximum likelihood logistic regression in R: all functions.

Maximum likelihood logistic regression in Matlab: estimation, likelihood, prediction.

Fudged functions needed for the old Matlab on CDF: fudged estimation, fudged minus log likeihood.

Solution: R functions for ordinary LR, R functions for bounded LR, R test script, output of script, plots produced by script.Handout: ps, pdf.Assignment 3:

Data files: train x, train y, test x, test y.

Solution: MLP functions, script, output, plots.Handout: ps, pdf.Assignment 4:

Here are some notes on how to do some things you'll need to do.

Here is the data.Note:I mistakenly put only 49 cases in the data file. That's OK. Just use 49 rather than 50.

Solution: Main functions, script, output plots.Handout: ps, pdf.

Here is the gene expression data and the cancer indicators.

**Demonstration programs**

Demo of K-NN and linear regression in R using prostate cancer data: knn program, script, training data, test data.

Demo of Naive Bayes in Matlab/Octave: nbayes program, script, training inputs, training targets, test inputs, test targets.

Demo of decision trees using the R "rpart" function: R script, output of script.

K means function and example: R function for K means, script to try it out.

Q learning program and demo: Q learning functions, Demo 1, Demo 2, Demo 2m.

These programs are also available on CDF in/u/radford/411/demo.

**Some useful on-line references**

Proceedings of the annual conference on Neural Information Processing Systems (NIPS)

Information Theory, Inference, and Learning Algorithms, by David MacKay

Reinforcement Learning: An Introduction, by Richard S. Sutton and Andrew G. BartoMy tutorial on Bayesian methods for machine learning: Postscript or PDF.

**Web pages for past related courses:**

STA 414 (Spring 2006)

CSC 321 (Spring 2006, Geoffrey Hinton)

CSC 411 (Fall 2005, Anthony Bonner)

CSC 411 (Fall 2004, Richard Zemel)