15-387/86-375/675 Computational Perception
Carnegie Mellon University
The perceptual capabilities of even the simplest biological organisms are far beyond what we can achieve with machines. Whether you look at sensitivity, robustness, adaptability and generalizability, perception in biology just works, and works in complex, ever changing environments, and can make inference on the most subtle sensory patterns. Is it the neural hardware? Does the brain use a fundamentally different algorithm? What can we learn from biological systems and human perception?
In this course, we will study the biological and psychological data of biological perceptual systems, mostly the visual system, in depth, and then apply computational thinking to investigate the principles and mechanisms underlying natural perception.
You will learn how to reason scientifically and computationally about problems and issues in perception, how to extract the essential computational properties of those abstract ideas, and finally how to convert these into explicit mathematical models and computational algorithms. The course is targeted to both neuroscience and psychology students who are interested in learning computational thinking, as well as computer science and engineering students who are interested in learning more about the neural and computational basis of perception. Prerequisites: First year college calculus, differential equations, linear algebra, basic probability theory and statistical inference, and programming experience are desirable.
All Office Hours will be held on zoom, using course zoom link unless notified and arranged otherwise
|Instructors ||Office Hours. ||Email (Phone)
|Tai Sing Lee (Professor) || Friday 9:00 am. Zoom Office Hour ||firstname.lastname@example.org
|Tianqin Li (TA) || Monday/Tuesday 7:00-8:00 p.m. || email@example.com
- Class location and time: WEH 4708. Monday/Wednesday 12:30 p.m - 1:50 p.m.
- Class recitation and journal club: Class Zoom. Friday 12:30 p.m - 1:50 p.m.
- Website: http://www.ni.cmu.edu/~tai/cp23.html (course info and readings )
- Canvas: Lecture materials and Information would be on Canvas.
- Handouts on Canvas .
- Frisby and Stone Seeing: The computational approach to biological vision . MIT Press, 2010 (recommended).
- Refrain from using laptop and cell phones during class.
Grading Scheme 15-387/86-375
|Evaluation||Grade Points |
|Assignments || 60 |
|Midterm ||10 |
|Final Exam ||20 |
|Class Participation ||10 |
- Total points: 100
- Grading scheme: A: > 88, B: > 75. C: > 65. D > 50.
Grading Scheme 86-675
|Assignments || 60 |
|Midterm || 10 |
|Final Exam * || 20 |
|Journal Club * || 3 Presentations |
|Term Project * || See below. |
|Class participation || 10 |
- Total credit for 86-675: 100
- Journal Club at least 10 times attendance, 2-3 presentations.
- Term project or term paper can be used to replace the journal club.
- Grading scheme: A > 88, B: > 75. C: > 65
- There will be 5 homework assignments involving Pytorch. The focus is on performing
experiments and analysis on existing implementation of perceptual model rather than
coding algorithms from scratch.
- Each student will have 7 days grace period for late homework. This grace period
can be used for one or multiple assignments. Use it wisely and
you cannot ask for more.
CANVAS or gradescope submission time relative to deadline will be used to track. If you are late by one hour, you will still be counted as one day.
- You can do a term project to replace one of the homework.
A term project must involve some computational experiments, using either downloadable softwares or programs you develop, as well as a conference-style paper.
All the codes should be documented and archived in github and submitted together with a term paper
(4 pages + additional pages for references) a pdf/doc file and/or matlab zip files.
Term project should take about at least 20 hours
to complete. Undergraduate student should work on his/her own project.
Graduate student planning to use the term project to replace the journal club should also work alone.
Graduate students can work on a two-person team on a larger scale project
but need to be approved in advance.
The term paper should cover background literature, and should adopt the format of a conference paper.
Project proposal is due one month before the end of the semester, but students
are encouraged to discuss project ideas with the professor
earlier on in the semester.
- Journal club option is an available option for 86-675 students.
Each student is expected to present two to three times during the course of the semester and attended 90% of the journal club.
- There will be a midterm (10 points) and a final exam (20 points) to test materials covered
in the lectures and homework assignments.
- There would be at least 2 in-class exercises given out in random time, each worths 1 point. Full credit: 10 points.
| ||SENSORY CODING || || |
|M 8/28 ||1. Introduction || || |
|W 8/30 ||2. Perceputal Theories || || |
|M 9/4 ||Label Day (no class) || || |
|W 9/6 ||3. Sensors and Retina ||
|| Homework 1 |
|M 9/11 ||4. Frequency Analysis || || |
|W 9/13 ||5. Pyramid || || |
|M 9/18 ||6. Computation || ||   |
|W 9/20 ||7. Retinex || || Homework 2 |
| ||PERCEPTUAL INFERENCE || || |
|M 9/25 ||8. Intrinsic Images || || |
|W 9/27 ||9. Shape and Motion || || |
|M 10/2 ||10. Visual Cortex || || Mid-Course Evaluation |
|W 10/4 ||11. Networks || || Homework 3 |
|M 10/9 ||12. Contours || || |
|W 10/11 || Midterm || || |
|F 10/13 || Family Weekend || || |
|M 10/16 || Fall break || || |
|W 10/18 || Fall break || || |
|F 10/20 || Fall break || || |
|M 10/23 ||13. Junctions || || Mid-term Grade. Project Proposal due |
|W 10/25 ||14. Organization || || |
|F 10/27 || Community Day - No Class ? || || |
|M 10/39 || 15. Texture || || |
|W 11/2 || 16. Metamers || ||Homework 4 (Nov 5) |
|M 11/6 || 17. Surfaces || || |
|W 11/8 || 18. Objects || || |
|M 11/13 || 19. Scenes || || |
|W 11/15 || 20. Synthesis || || Proposal due. Homework 5 (out Nov 18) |
|M 11/20 || 21. Integration || || |
|W 11/22 ||Thanksgiving break || || |
|M 11/27 ||22. Composition || || |
|W 11/29 ||22. Attention || || |
|M 12/4 ||23 Cognition || || HW 5 due |
|W 12/6 ||24. Review || || |
|S 12/X ||Final Exam TBA || || |
Reading (relevant, but optional reading)
Week 1 (Lectures 1 and 2) Observations, Theories and Computational Philosophy
Week 2,3 (Lectures 3, 4, 5, 6) Retina, Frequency, Pyramid and Computation
- Visual perception starts with the eyes and the photoreceptors. However,
there is already sophisticated computation in the
retina. We will read some classic and the modern papers on retinal processing, cover some basic background on
frequency analysis, pyramid representation, as well as the current computational approach (via deep learning) for modeling
retinal processing. We will do a problem set on retinal processing, and explore its relationship to some visual illusion
Lettvin, Maturana, McCulloch and Pitt. (1959) What the frog's eye tells the frog's brain Proceedings of the IRE 1940-1959 .
Gollisch and Meister (2010) Eye Smarter than Scientists Believed:
Neural Computations in Circuits of the Retina Neuron 65: 151-164.
Maheswaranthan, .... Ganguli and Baccus (2018) Deep learning models reveal internal structure and diverse computations in the retina under natural scenes bioRxiv, June 8, 2018.
McIntosh, Maheswaranathan, Nayebi, Ganguli and Baccus (2016) Deep Learning Models of the Retinal Response to Natural Scenes NIPS
Burt and Adelson (1983) The Laplacian Pyramid as a Compact Image Code. IEEE Transaction on Communications. Com-31, no 4. 532-540
Week 4 (Lecture 7,8) Lightness perception and Intrinsic Images
- Our perception of brightness (or lightness) and color is not determined what are sensed by the retina, but in fact an interpretation of the ligthness and color properties of the object surfaces in the world. We will explore the classic theory of retinex as well as modern computational theory of intrinsic images for understanding lightness and color perception, culminating in a problem set on these issues.
Adelson, Ed, (2000) Lightness Perception and Lightness Illusion The New Cognitive Neuroscience, Gazzaniga ed. MIT Press.
Land, E, (1977) The retinex theory of color vision Scientific America
Horn, B, (1974) Determininng lightness from an image Computer Graphics and Image Procwssing.
Morel JM, Petro AB, Sbert C. (2010) A PDE Formalization of Retinex Theory IEEE Trans Image Process.
Tappen, Freeman and Adelson (2005) Recovering intrinsic images from a single image IEEE PAMI.
Michael Janner, Jiajun Wu, Tejas D. Kulkarni, Ilker Yildirim, Joshua B. Tenenbaum (2017) Self-Supervised Intrinsic Image Decomposition. NeurIPS.
Wei-Chiu Ma, Hang Chu, Bolei Zhou,
Raquel Urtasun and Antonio Torralba1 (2018) Single Image Intrinsic Decomposition wihtout a Single Intrinsic Image. ECCV
Week 5 (Lecture 9,10). Surfaces, Shapes and Visual Cortex
- Shading as well as many other cues allow us to infer 3D shapes. Intrinsic images of shading is a consequence of illumination on 3D shape, suggesting illumination and 3D shapes are further decompositon of shading images. We wonder whether how 3D shape as an intrinsic image or objects is represented in the brain. Is it 2D images, 2.5D sketches or 3D solids.
Ramaujan Srinath, A. Emonds, O. Wang, A. Lempel, E. Dunn-Weiss, CE, Connor, K. Nielsen
Early Emergence of Solid Shape Coding in Natural
and Deep Network Vision. Current Biology, 31, 51-65. 2021.
S Y Edelman, M J Tarr on "How are three-dimensional objects represented in the brain?"
Cerebral Cortex 1995 May-Jun;5(3):247-60.
Zhang, X, Zhang, Z, Zhang C, Tenenbaum J, Freeman W, Wu, J. Learning to Reconstruct Shapes from Unseen Classes NeurIPS 2018
Week 6, 7 (Lecture 11, 12, 13). Perceptual Learning and Inference
Week 8,9 (Lectures 14, 15, 16, 17) Perceptual Organization and Segmentation
- In addition to inferring "visible" physical properties of the world, the brain also tries to organize the sensory information into parsimonious descriptions to infer more abstract and global properties or summary statistics of the world such as boundary, and surface properties or summary statistics.
In this segment of the course, we will study Gestalt school of thoughts and models for extracting these properties.
Bela Julez (1981) Textons, the elements of texture perception and their interaction. Nature 290. 91-97.
Heeger and Bergen (1995) Pyramid-based texture analysis/synthesis SIGGRAPH 1995
Portilla and Simoncelli (2000) A parametric texture model based on joint statsitics of complex wavelet coefficients Internal journal of computer vision 40(1), 49-71.
L. A. Gatys, A. S. Ecker, and M. Bethge (2015) Texture Synthesis Using Convolutional Neural Networks NIPS 28
L. A. Gatys, A. S. Ecker, and M. Bethge (2016)
Image Style Transfer Using Convolutional Neural Networks CVPR 2016
Freeman J, Simoncelli EP. (2011) Metamers of the ventral stream. Nature Neuroscience.
Kovacs, I., Papathomas, T., Yang, M. and Feher, A. (1996). When the brain changes its mind: Interocular grouping during
binocular rivalry. Proceedings of the National Academy of Sciences, 93(26), pp.15508-15511.
- Lee, T.S. (1995). A Bayesian framework for understanding texture segmentation in the primary visual
cortex. Vision Research 35, 2643-2657..
- Tsao and Tsao (2022) A topological solution to object segmentation and tracking.
Week 10. (Lecture 18, 19) Objects, Scenes and Inverse Graphics
- Early computer vision approach emphasized on analysis by synthesis. This framework can be generalized to conceptualize the recurrent interaction in the hierarchical visual system and perception as inverse graphics. We will study these theories and their neural foundation, as well as to see how these principles can be extended to self-supervised learning based on prediction principles.
Van Essen, Anderson and Felleman (1992) Information processing in primate visual systems: an integrated approach Science 5043: 419-423.
Mumford, D (1992) On the computational architecture of the neocortex Biological Cybern.
Torralba and Oliva
(2003) Statistics of natural image categories Network: Comptuations in Neural Systems 14: 391-412.
Rao and Ballard (1998) Predictive coding in the visual cortex: a functional interpretation of some oextra-classical receptive field effects Nature Neuroscience
Lee and Mumford (2003) Hierarchical Bayesian inference in the visual system J. Optical Society of America
Lee, T.S. (2015) The Visual System's Internal Models of the World Proceedings of the IEEE Vol 103, issue 8, 1359-1378.
Lotter, Krieman and Cox (2020) A neural network trained for prediction mimics diverse features of bioloigcal neurons and perception Nature Machine intelligence
vol 2, 210-219.
Ilker Yildirim, Mario Belledonne, Winrich Freiwald, Josh Tenenbaum (2020) Efficient inverse graphics in biological face processing Sci. Adv. 2020; 6 : eaax5979 4 March 2020
T. D. Kulkarni, W. F. Whitney, P. kohli, J. Tenenbaum, Deep convolutional inverse graphics network, in Proceeding of the Advances in Neural Information Processing Systems (NIPS, 2015), pp. 2539â€“2547
Kar K, Kubilius J, Schmidt K, Issa EB, DiCarlo JJ. Evidence that recurrent circuits are critical to the ventral stream's execution of core object recognition behavior. Nature Neuroscience. 2019. doi: 10.1038/s41593-019-0392-5.
Week 11 (Lecture 21, 22) Integration and Composition
- Compositionary Theory argues the brain or the visual system is organized in a recursive nearly decompositional system that best models the compositional nature of parts, objects and scenes in the natural world, their flexible combination and deformation. We will explore some classical and modern theories on composition, exploring the linkage between modern deep neural networks and symbolic AI from this perspective, and the connection between language and vision.
E. Bienenstock, S. Geman, and D. Potter. Compositionality, MDL Priors, and Object Recognition NIPS Advances in Neural Information Processing Systems 9.
S. Geman, Hierarchy in machine and natural vision. Proceedings of the 11th Scandinavian Conference on Image Analysis,
Yuille. Towards a Theory of Compositional Learning and Encoding of Objects 1st IEEE Workshop in Information Theory in Computer Vision and Pattern Recognition. ICCV
S.C. Zhu and D. Mumford (2006) A Stochastic Grammar of Images Foundations and Trends in Computer Graphics and Vision 2(4): 259-362.
Week 12 (Lecture 23, 24) Attention, Cognition and Consciouness
Additional Exploration: Art and Beauty
What is beauty? Is it just something in the eyes of the beholders, or is it something universal? In this final week of the class, we hope to explore the concepts and computational theories of beauty and art and how they might be related to art, and the computational principles underlying
perception that we have studied in the course.
Cavanagh, P. (2005) The artist as neuroscientist. Nature, 434, 301-307.
Perdreau, F. & Cavanagh, P. (2011). Do artists see their retinas? Frontiers in Human Neuroscience, 5:171
Bilge Sayim and Patrick Cavangah (2011) What line drawings reveal about the visual brain.
L. A. Gatys, A. S. Ecker, and M. Bethge (2017) Texture and art with deep neural networks Current Opinions in Neurbiology 46, 178-186.
Schmidhuber Jurgen. (1997) Low complexity art.
Schmidhuber Jurgen. (2008) Driven by Compression Progress: A simple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, and jokes!
Questions or comments:
contact Tai Sing Lee
Last modified: August 2023, Tai Sing Lee