15-387/86-375/675 Computational Perception

Carnegie Mellon University

Fall 2022

Course Description

The perceptual capabilities of even the simplest biological organisms are far beyond what we can achieve with machines. Whether you look at sensitivity, robustness, adaptability and generalizability, perception in biology just works, and works in complex, ever changing environments, and can make inference on the most subtle sensory patterns. Is it the neural hardware? Does the brain use a fundamentally different algorithm? What can we learn from biological systems and human perception?

In this course, we will study the biological and psychological data of biological perceptual systems, mostly the visual system, in depth, and then apply computational thinking to investigate the principles and mechanisms underlying natural perception. You will learn how to reason scientifically and computationally about problems and issues in perception, how to extract the essential computational properties of those abstract ideas, and finally how to convert these into explicit mathematical models and computational algorithms. The course is targeted to both neuroscience and psychology students who are interested in learning computational thinking, as well as computer science and engineering students who are interested in learning more about the neural and computational basis of perception. Prerequisites: First year college calculus, differential equations, linear algebra, basic probability theory and statistical inference, and programming experience are desirable.

Course Information

Instructors Office Hours. Email (Phone)
Tai Sing Lee (Professor) Friday 10:00 am. Zoom Office Hour taislee@andrew.cmu.edu
Shang Gao Li (TA) Tuesday 8:00 p.m. and Friday 5:00 p.m. shanggao@andrew.cmu.edu
Violet Han (half-TA) Monday 5:00 p.m. yinuoh@andrew.cmu.edu
  • All Office Hours will be held on zoom, using course zoom link unless notified and arranged otherwise
  • Recommended Textbook

    Classroom Etiquette

    Grading Scheme 15-387

    EvaluationGrade Points
    Assignments 65
    Midterm 10
    Final Exam 15
    Class Participation 10

    Grading Scheme 86-375

    Evaluation Points
    Assignments 39
    Midterm 10
    Final Exam 15
    Flex Requirement 26
    Class participation (10) 10

    Grading Scheme 86-675

    Assignments 65
    Midterm 10
    Final Exam * 15
    Journal Club * 3 Presentations
    Term Project * See below.
    Class participation 10


    Term Project

    Term Paper

    Journal Club



    Date Lecture Topic Assignments
    M 8/29 1. Introduction    
    W 8/31 2. Perceputal Theories    
    M 9/5 Label Day (no class)    
    W 9/7 3. Sensors and Retina   Homework 1
    M 9/12 4. Frequency Analysis  
    W 9/14 5. Pyramid    
    M 9/19 6. Lightness and Color    
    W 9/21 7. Retinex and Intrinsic Images   Homework 2
    M 9/26 8. Lightness perception    
    W 9/28 9. Shape from Shading    
    M 10/3 10. Visual Cortex   Mid-Course Evaluation
    W 10/5 11. Neural Networks   Homework 3
    M 10/10 12. Contours    
    W 10/12 Midterm    
    F 10/14 Family Weekend    
    M 10/17 Fall break    
    W 10/19 Fall break    
    F 10/21 Fall break    
    M 10/24 13. Junctions   Mid-term Grade. Project Proposal due
    W 10/26 14. Efficient Code    
    F 10/28 Community Day - No Class    
    M 11/1 14. Organization    
    W 11/3 15. Texture   Homework 4 (Nov 5)
    M 11/7 16. Metamers    
    W 11/9 17. Surfaces    
    M 11/14 18. Objects    
    W 11/16 19. Inferences   Proposal due. Homework 5 (out Nov 18)
    M 11/21 20. Composition and Objects    
    W 11/23 Thanksgiving break    
    M 11/28 21. Attention    
    W 11/30 22. Consciousness and Cognition    
    M 12/5 23 Review   HW 5 due
    W 12/7 In-class Final Exam   Take-home due
    S 12/18 Final Exam Slot. 8:30-11:30 am    

    Reading (relevant, but optional reading)

    Week 1 (Lectures 1 and 2) Observations, Theories and Computational Philosophy

    Week 2,3 (Lectures 3, 4, 5) Retina, Pyramid and Neural Network

    Week 4 (Lecture 6,7,8) Lightness perception and Intrinsic Images

    Week 5 (Lecture 9,10). Perception of 3D shapes

    Week 5 (Lecture 9,10). Source Separation and Representation Learning

    Week 6 (Lectures 11, 12) Perceptual inference: contours, depth, surfaces

    Weeks 7 and 8 (Lectures 13, 14, 15, 16) Perceptual Organization: Grouping, Gestalt, Content and Style

    Week 9 and 10. (Lecture 17, 18, 19, 20) Analysis by Synthesis and Predictive Coding

    Week 11 (Lecture 21, 22) Compositional Theory, Scenes, Objects and Parts

    Week 12 (Lecture 23, 24) Attention and Awareness

    Week 13 (Lecture 25, 26) Perception, Art and Beauty and Review

    Questions or comments: contact Tai Sing Lee
    Last modified: August 2022, Tai Sing Lee