15-387/86-375/675 Computational Perception

Carnegie Mellon University

Fall 2021

Course Description

The perceptual capabilities of even the simplest biological organisms are far beyond what we can achieve with machines. Whether you look at sensitivity, robustness, adaptability and generalizability, perception in biology just works, and works in complex, ever changing environments, and can make inference on the most subtle sensory patterns. Is it the neural hardware? Does the brain use a fundamentally different algorithm? What can we learn from biological systems and human perception?

In this course, we will study the biological and psychological data of biological perceptual systems, mostly the visual system, in depth, and then apply computational thinking to investigate the principles and mechanisms underlying natural perception. You will learn how to reason scientifically and computationally about problems and issues in perception, how to extract the essential computational properties of those abstract ideas, and finally how to convert these into explicit mathematical models and computational algorithms. The course is targeted to both neuroscience and psychology students who are interested in learning computational thinking, as well as computer science and engineering students who are interested in learning more about the neural and computational basis of perception. Prerequisites: First year college calculus, differential equations, linear algebra, basic probability theory and statistical inference, and programming experience (Matlab) are desirable.

Course Information

Instructors Office or zoom (Office hours) Email (Phone)
Tai Sing Lee (Professor) Friday 9-10 a.m. Class zoom link taislee@andrew.cmu.edu
Tianqin Li (TA) Monday 5:30-6:30 pm and Tuesday 8:00-9:00 p.m. Class zoom link tianqinl@cs.cmu.edu

Recommended Textbook

Classroom Etiquette

Grading Scheme 15-387

EvaluationGrade Points
Assignments 65
Midterm 10
Final Exam 15
Class Participation 10

Grading Scheme 86-375

Evaluation Points
Assignments 39
Midterm 10
Final Exam 15
Flex Requirement 26
Class participation (10) 10

Grading Scheme 86-675

Assignments 65
Midterm 10
Final Exam * 15
Journal Club * 3 Presentations
Term Project * See below.
Class participation 10


Term Project

Term Paper

Journal Club



Date Lecture Topic Assignments
M 8/30 1. Introduction    
W 9/1 2. Computational Approach    
M 9/6 Label Day (no class)    
W 9/8 3. Retina   Homework 1
M 9/13 4. Frequency Analysis  
W 9/15 5. Neural Network    
M 9/20 6. Optics, Lightness and Color    
W 9/22 7. Retinex and Intrinsic Images   Homework 2
M 9/27 8. Lightness perception    
W 9/29 9. Dimensional reduction    
M 10/4 10. Source Separation   Mid-Course Evaluation
W 10/6 11. Belief Net   Homework 3;
M 10/11 12. Inference: Depth    
W 10/13 Midterm    
Th 10/14 Mid-semester break    
M 10/18 13. Inference: Motion   Mid-term Grade. Project Proposal due
W 10/20 14. Perceptual Organization    
M 10/25 15. Texture Perception    
W 10/27 16. Content and Style   Homework 4
M 11/1 17. Visual Hierarchy    
W 11/3 18. Analysis by Synthesis    
F 11/5 No Journal Club: Community Engagement    
M 11/8 19. Predictive coding    
W 11/10 20. Self-supervised learning   Homework 5
M 11/15 21. Compositional theory    
W 11/17 22. Object and Parts    
M 11/22 23. Attention    
W 11/24 Thanksgiving break    
M 11/29 24. Awareness   HW 5 due
W 12/1 25. Review    
F 12/3 Last day of Class   Paper Presentation
X 12/18 Final Exam and Presentations    

Reading (draft)

Week 1 (Lectures 1 and 2) Computational Philosophy

Week 2,3 (Lectures 3, 4, 5) Retina and Neural Network

Week 4 (Lecture 6,7) Lightness perception and Intrinsic Images

Weeks 5 and 6 (Lecture 8,9,10,11). Source Separation and Dimensional Reduction

Week 7 (Lectures 12-13) Perceptual inference: depth and motion

Week 8 (Lectures 14-16) Perceptual Inference: Grouping, Organization and Style

Week 9 and 10. (Lecture 17-20) Analysis by Synthesis and Predictive Coding

Week 11 (Lecture 21, 22) Compositional Theory, Objects and Parts

Week 12 (Lecture 23, 24) Attention and Awareness

Questions or comments: contact Tai Sing Lee
Last modified: August 2021, Tai Sing Lee