Effects of Foveation on Early Visual Processing

Billy Broderick

April 14, 2022

Receptive fields grow with eccentricity

… and as you go up the visual hierarchy

Receptive fields grow linearly with eccentricity

Visual system

Outline

Spatial frequency

spatial frequency is roughly equivalent to the speed of change in an image and is important for characterizing V1 neuronal responses. it’s often studied using 2d sine wave gratings like these, which drive responses in V1 very well

here, the gratings go from low frequency (slow changes) on the left to higher and higher as you move to the right

I should note, spatial frequency is more than just some abstract property of gratings – any image can be decomposed into sine gratings like these using a Fourier transform. that is, we can create any image that could possibly exist from a weighted sum of gratings like these. thus, if we know how neurons respond to gratings like these, we can, in principal, predict how they’ll respond to any image (though in practice, that prediction will be missing some important aspects)

Macaque V1 neurons are tuned for spatial frequency

This tuning changes with eccentricity

Stimulus period grows linearly with eccentricity

Stimuli span a wide range of eccentricities

… and orientations

fMRI responses are tuned for spatial frequency

Peak spatial frequency gets higher as eccentricity drops

This holds across stimulus classes, though peak spatial frequency may differ

Model parameters

vertical vs horizontal. there’s lots of evidence in the perceptual literature about differences between sensitivity to vertical and horizontal information, so we wanted to allow for our model to capture the possibility that the preferred period was higher for vertical than horizontal orientations, or vice versa.
note that we don’t actually show any horizontal or vertical stimuli. but, for each voxel, some of the stimuli we show will be vertical and some will be horizontal. with this stimulus, for example, a voxel along the horizontal meridian will see a vertical orientation, while a voxel along the vertical will see a horizontal orientation. for this voxel along the horizontal meridian, tangential always means vertical, but by fitting all our voxels across our whole stimulus set, we can fit these two effects separately

Model parameters

Largest effect is eccentricity dependence, others are modest

we used cross-validation to see which of these parameters were necessary to fit our data

there’s a lot going on here, but I’ll walk you through it.

to orient you: each point on the right is a model fit with a different combination of parameters, going from the fewest parameters on the top to the most at the bottom. the x-axis here is the loss, so lower is better. we fit each subject separately and then combined their losses to get the points here. the matrix on the left here tells you which parameters were fit in each model (the others were set to 0): the colored blocks say which parameters were enabled

the first three models are testing different possibilities for the effect of eccentricity on preferred period, then beyond that we start looking at the effects of orientation on preferred period and relative amplitude.

Largest effect is eccentricity dependence, others are modest

but we get an even bigger improvement when we fit both the slope and the intercept, allowing for preferred period to be an affine function of eccentricity

this improvement is the biggest effect by far, all the other improvements are fairly modest

beyond these first three, every set of parameters improves performance except for the two of the amplitude praameters, as shown in the pale dots.

now, the effect of each set of parameters is relatively small, but model 9 is definitely an improvement on model 3, and this is consistent across subjects.

so we’re that the preferred period and relative amplitude are modulated by orientation, but the largest effect is this affine effect of eccentricity

I included this figure and the parameters in my paper, as well as the code for my model and csv of these values on my github, so researchers could decide for themselves whether their research questions require taking all these subtleties into account

and so what does that mean?

Preferred period is an affine function of eccentricity

you can read the paper or ask me questions later if you want all the gorey details, but I’ll go through one important result

I’m plotting here the preferred period as a function of eccentricity for these two classes of stimuli, averaged over all directions in the visual field. the shaded region here is the 68% confidence interval bootstrapped across subjects

we see that, like I said, preferred period is an affine function: it’s a line with a non-zero intercept. that’s worth pointing out because there’s generally a question of what happens near the fovea, for this and related properties, like the size of receptive fields. the preferred period can’t actually go to zero, which would imply that the peak spatial frequency was infinite (and so you’d have infinite resolution), but it could go to zero and then level off somewhere.

we find, pretty strong evidence that that does not happen: the intercept is instead somethign nonzero.

Size of offset equivalent to 3 degrees eccentricity

Effect of orientation largest at horizontal meridian, disappears at vertical

and cancel on the vertical meridian, where annuli stimuli are horizontal

what I’m showing you here is a single example of the effect of orientation that we found, there are others that we go explore in more detail in the paper

(this suggests that the quality of spatial representation depends on polar angle, which is consistent with a large body of psychophysical results, which generally show better performance along the horizontal meridian than the vertical)

(possible reason: if we’re walking through a static scene, objects are going to move radially as we approach them. objects moving tangentially however, would either mean that they’re static and we’re moving up and down, or they’re moving independently of us, which might be more important to note. more work is needed on this)

Intermediate conclusions

Developed novel set of scaled grating stimuli
Measured voxel spatial frequency tuning across human V1
Fit responses of all voxels simultaneously with a single 9 parameter model
Showed an affine relationship between preferred period and eccentricity
Showed effects of orientation on preferred period and relative amplitude
Shared data, parameters, and code: https://github.com/billbrod/spatial-frequency-preferences
Serves as a step towards a generalized model of whole map

So this work does… read points

that point about the general map is worth expanding on a bit. I mentioned briefly in the beginning that this project came about because I wanted to know what this relationship looks like in order to build a model of all of v1 in fMRI

V1 is one of the best characterized areas of the primate central nervous system, and we have standard models that have been around for a while. we should be able to do something like my initial goal – build a model that predicts responses to an arbitrary stimulus across all of V1 simultaneously. that model will undoubtably fail in spectacular ways, but even getting something somewhat sensible together to test would be very useful. this work provides one element that is necessary for such a model of the whole map, by summarizing spatial frequency tuning, an improtant response property of V1 neurons, at the level of detail we think we need for such a model.

I didn’t build that model, however. so that’s a good project idea for any of the grad students in the audience

Perceptual metamers

To do that, we’re using metamer, which are physically distinct stimuli that are perceptually identical. they’re an old concept in vision science, dating back to the color-matching experiments of Helmholtz in the first half of the 19th century, where they were integral to the development of the trichromatic theory of human color vision, which provided evidence for 3 cone classes more than a hundred years before any physiological evidence.

these color metamers are patches that produced different wavelengths of light but resulted in the same perceptual color

for example, when you look at my ?? shirt in the corner of your Zoom window, that may look the same as if you saw this shirt in person, but the actual light that enters your eye is very physically different, because your computer only has 3 color primaries: the amount of energy at each wavelength is far from the same. they look so similar because your visual system has thrown out the information you could use to distinguish them. Note that in this case, that happens in the very beginning of the visual system, with the cones throwing away information about the wavelength of light, but this discarding of information happens throughout your visual system.

in particular, I’m going to look at how spatial information is discarded

Project overview

Built foveated models of the early visual system
Created hundreds of model metamers
Showed them to humans in psychophysics experiment
Found largest model parameter whose model metamers are also human metamers

Local average luminance model

the simplest foveated model we can build will just average the local luminance across the image. that is, we’re just finding how bright each region of the image is

we built a set of log Gaussian windows whose width grows linearly with eccentricity and we take the average pixel value in each of those windows. this here is a schematic, showing the contour at half max amplitude – in actuality, the windows overlap quite a lot in order to uniformly cover the image

with these size windows, with this image, the average is pretty similar in the top part of the image, slightly darker at the bottom fo the palm tree.

those average pixel values are all the model knows about the image. according to this model, any image that has the same average luminance in those windows, they’re all the same. doesn’t matter if they have the same average because they have one bright pixel and a bunch of dark ones, all the same value, or it’s a sine wave oscillating around the mean: as long as the average pixel values match, they’re identical

These models have a single parameter: scaling

that’s our first model, and it’s a simple one. it has a single parameter we’re varying, which is how big these windows are: like I said, these window widths grow linearly, so the slope of the line relating their size to their eccentricity is that parameter, which we call scaling.

this is similar to the scaling idea of the last chapter, in that it’s a linear increase with eccentricity. previously, it was of the preferred period, here it’s the size of these pooling windows, which model neuronal receptive fields

for these two sets of windows, for example, the one on the left has a smaller scaling, thus smaller windows and more windows, and the one on the right has a larger one, thus larger windows and fewer windows

that means the one on the right throws away more info than the one on the left. you could think of its representation being “blurrier” or “lossier” – it will consider a larger number of images to be identical than the one on the left

that’s fairly abstract, so let’s see what that looks like

this is that same image, but with the contour of the window at half-max amplitude shown, to give you a sense for how large the regions in which we’re averaging the pixel values is

to reiterate, what we have here is a model metamer for reasonably large windows. but it’s not a human metamer: while the model with this size windows thinks this image is identical to the target, participants in our experiment can easily tell the difference between the two

the goal of this project is to find the largest window size, the largest scaling value where the model metamers are human metamers.

it’s also worth looking at this in a bit of detail. you might look at this and say – this image is weird. it’s somehow both staticky and blurry, but based on that description you just gave me, I was expecting it to just be blurry

and you’re not wrong – as our models windows get big towards the edge of the image, the constraitn to just match the local luminance is pretty loose and, in particular, it is insensitive to high frequencies, which show up as this kind of salt-and-pepper noise you see here.

because it’s insensitive to them, it doesn’t see them at all! that means that we can do anything we like in those high frequencies and the model won’t be able to tell the difference. in this example, there’s a lot of high frequencies, which are random or scattered.

we could equally have thrown all the high frequencies out, just resulting in a blurry image. those would both be model metamers to the target image

this is the usefulness of the metamer paradigm, that it creates images that are a bit surprising, helping us to explore the space of all possible images in a way that helps us find informative ones

Local spectral energy model

we built a second model, an energy model. like the previous model, we’re pooling an image statistic in windows that grow larger with eccentricity. whereas that model was pooling pixel values, this builds off the V1 complex cell model that I mentioned at the beginning of my defense, pooling the spectral energy at 4 orientations and 6 spatial frequency bands.

this means that instead of pooling the local luminance, we’re pooling how much “stuff” there is at each orientation and spatial frequency, and we think this is similar to what V1 is doing. this model is built directly on top of the image, not on the outputs of the last model, and it includes the luminance as well

the first model was blurring pixel values, whereas this one is blurring complex cell outputs (which are themselves blurred simple cell outputs), for those who are used to thinking like that. and we think that there will be some relationship between identity of the statistics and how big the pooling regions are

More complex statistics can be pooled over a larger area

Distortions depend on which statistic is being pooled

Interaction between statistic and window size is what matters

Experiment structure

Predictions

now, what we had thought, based on previous work, is that the critical scaling value should roughly match up with the scaling measured from physiology, that is, the scaling of neuronal receptive fields in the corresponding brain area. so, maybe the retina for the luminance model and V1 for the energy
the bars are showing approximately what the physiological scaling values are, for V1 and for midget and parasol cells, the two most common ganglion cell classes in the retina
The idea is that this calculating of summary statistics within receptive fields is roughly what neurons in the visual system are doing. so if you look at a model metamer generated with a larger scaling value than the corresponding human brain area, then you should be able to notice the difference, because the model is throwing away information that you’re still sensitive to. if the windows are smaller, however, then you should not notice, because your brain is throwing away more information than the model.
the reality is a bit more complicated

Data

so here’s the data we got when participants compared metamers to original images, the natural images that were the target of synthesis. note that the x-axis is log-scaled.

still showing the bars from before

so:

the data is lawful and monotonic, fit reasonably well by curves, goes pretty much from chance at 50% to ceiling at 100%
critical scaling differ from each other by about 4x.
our luminance results (.016) are between midget and parasol (.01 and .03) cells’ physiological scaling, energy (.06) is half the bottom edge of V1 distribution (.25)

so we are seeing that, as the statistics get more complex, the spatial scale of pooling grows as well. this aligns with our sense of what happens in the visual system, but the numbers for the energy model differ substantially from V1. this bar comes from macaque data and humans have higher acuity than macaques, so it’s possible that the numbers are smaller for humans (RGC bars come from human data). it could also mean that our model is missing something key about V1 computation

we ran some additional variations on this to try and get at this question a bit, which I don’t hvae time to discuss here. let me know if you’re interested and we can discuss it offline

physiology

previous work surveying the literature found critical scaling in V1 was .2 to .3
looking at some Dacey papers from the early 90s, midget critical scaling is about .01, parasol is ~.03
RGC is first area where you get linear relationship between size and eccentricity

if someone asks about size of shaded region: that’s the distribution of the posterior means across subjects and images. if we look at those separately…

Parameters

Intermediate conclusions

Built foveated models of two stages of early visual processing
Synthesized large set of model metamers: https://users.flatironinstitute.org/~wbroderick/metamers/
Ran psychophysics experiment to find each model’s critical scaling
Showed spatial scale of pooling grows with statistic complexity

So this work does… read points

let’s look at this website a bit. this was put together by several engineers at the Flatiron Institute, Liz Lovero, Dylan Simons (also a C+P alum), and Aaron Watters. it has a brief description fo the project, and then shows the target image and metamers side by side, with a zoom in panel. you can select any of the images to view, and filter or sort by the different categories. in this way, you can see all the metamers I generated for the project, not just the ones I highlight in the paper or presentation itself. and, though I haven’t done it yet, we’re going to add a link to bulk download them all

I’ve spent a fair amount of time during my PhD thinking about and working on issues related to how to best share code and data related to my projects, which all fall under the category of…

Software is under-funded

… but heavily-used

92% of British academics use research software
69% say their research would not be practical without research software
Vast majority of top 100 all-time cited papers describe experimental methods or software
and that’s probably an undercount

but, by and large, we don’t. and that is, I think, because it’s hard! few of us are trained in software engineering, many students have no training in programming at all, and yet we spend a huge portion of PhDs writing and running code

and so we end up with code that works for our purposes, but is not suitable for others: maybe it can’t run on anyone else’s machine, maybe I’m not sure what its dependencies are, almost certainly it’s not well documented or with a readme that tells others how to use it

to be clear – I’m not trying to say that anyone who hasn’t taken the time to figure out how to do all this is a bad scientist and should feel bad. the reason they’re not doing it is, largely, because they don’t get trained on how to do it, they receive no feedback on it, and they’ll likely receive no reward for doing so. with researchers largely evaluated on publications and the faculty job market as tight as it is, it’s perfectly understandable that this falls to the wayside. what I’m trying to point out here is the incentives with which we have structured our scientific community are not working

plenoptic contents

Metamer: my second chapter, Freeman and Simoncelli 2011, Portilla and Simoncelli 2000
Eigendistortion: Berardino et al. 2017
Geodesics: Hénaff and Simoncelli 2015, Hénaff et al. 2019
MAD Competition: Wang and Simoncelli 2008
Models: Portilla-Simoncelli texture statistics, Steerable Pyramid, FrontEnd models from Berardino et al. 2017

these include several synthesis methods that have been developed over the past several decades, including metamers, which I discussed in the second part of my talk, as well as models, one of which provides the frontned of my energy model in that section

I don’t have time to describe these other methods, which span the theses of multiple students, in detail. but they all have a similar flavor to metamers: they synthesize new images to help better understand computational models. all of them were developed in the lab, but were used for basically a single project, so, like I mentioned before, the code was not meant for broader use

and, to be clear, what we’re doing here is not just reproducing these papers, but providing implementations that enable researchers to use these methods on their own models, enabling them to undrestand these models better

one of the factors that limited these methods’ application to other users was that all of these methods require differentiating the models involved, understanding how the output of the model changes as you alter the input. for a long time, this was incredibly difficult, because it meant you had to take the derivative of these models by hand, resulting in a lot of calculus, which you needed to repeat everytime you made a change to the model. this was SLOW and basically meant that the final implementation was so model-specific that it was nigh impossible to use on anything else

that allows the same code to support, for example, creating metamers with the models in my paper, VGG16 or other deep neural network implementations included in pytorch’s model zoo, and the classic Portilla-Simoncelli texture statistics, reimplemented from the original matlab release more than 20 years ago

here, I’m showing you metamers based on two different images, on the rows, for those four models I mentioned, on the column. the first two are the foveated models I presented in the last section, with a fairly large scaling value, and we can see similar patterns to what we saw before – increasing high frequency noise as we move away from the fovea for fov lum, increasing energy distortions as we move away for fov energy.

if you’re familiar with the portilla-simoncelli texture statistics, these will look familiar – they do a good job capturing the texturiness of this reptile pattern, so that this image looks pretty indistinguishable from the target, but fail completely to capture highly structured, like a portrait.

this last is from VGG16, a “classic” deep neural network developed for object recognition, matching the representation at the third max pooling layer, about halfway through the model. two thigns jump out: it’s insensitive to high frequency noise, like our fov lum model, though it’s convolutional and so this noise shows up everywhere in the image. second, it’s multicolored – it’s the only one of these models that operates on RGB images, but it does so in a way that shows it has very different color sensitivities than humans and, in particular, doesn’t recognize anything special about grayscale.

all of these metamers were generated using the same Metamer code object, I simply changed the inputs.

this allows researchers, both within and without of the lab, to focus on the scientific questions that interest them, instead of having to reinvent the wheel if they want to make use of these methods.

development moves in fits and starts, as it’s nobody’s main project, and so it’s not quite ready for broad usage yet, but we’re hoping to get to a version 1.0 release soon

Summary

Fit single model to all voxels in human V1 to show how spatial frequency preferences change across the visual field
Showed how spatial information is discarded by the early visual system
Discussed importance of open-source software
Described plenoptic, python package for models and synthesis methods

summarize whole thing:

in the first section, I demonstrated the usefulness of fitting all voxels at once for investigating how V1 spatial frequency preferences change across the visual field

in the second, I showed you what information is discarded by the early visual system, with models that average image statistics in pooling windows that grow as you move away from fixation

and in this final section, I described how open source software is critical but under-valued to academic science

and described plenoptic, a software library that a group of us developed to serve as a home for synthesis methods and models developed in the lab over the years, to serve as a form of institutional knowledge

and that’s it

(defense of open source:

what I’ve given in this talk was more of a general defense of software, rather than open-source software in particular, but I do think we should focus on open-source. first, similar caveat to earlier: I’m not saying if you don’t use open source you’re terrible or even that your work can’t be considered open. openness exists on many axes and you can still share your code, make sure it’s usable by others, etc while using MATLAB or other proprietary languages. and openness is not some pure binary, where if you fall short in one aspect you might as well not have tried, it’s a goal we’re striving for, so any step in that direction is worth taking.

that said, open source languages are better than proprietary. for this audience, that decision is generally python vs matlab, which is what I’m most familiar with, but similar arguments apply to other comparison. there are many reasons, access and licensing, and then ideological ones.

matlab costs money and so not everyone can use your code. the goal of open science is to make it as usable by others as possible. it’s true that most of the people in this room have access to matlab without having to pay for it but that’s because we’re in a wealthy university in a wealthy country. science is global, we should be encouraging that, not erecting boundaries. it is true that you can do things like run matlab code in octave or compile it so it can run without access to a license, but neither of those steps is trivial. I tried to do it for my spatial frequency project, which used a matlab toolbox to estimate the responses of the voxels, and couldn’t get it to work, so I gave up. if you think the fact that octave exists is sufficient, I encourage you to actually use octave to make sure your code works – it’s not simple
and to be clear, the cost isn’t nothing. we have absolutely no idea what matlab costs, because the universities and the companies are the consumers, not us. it’s similar to journals in that regard – there’s a complete disconnect between us as users and the cost. I was poking around online about this at one point, and I came across a directive from NASA – their advanced computing division only has 16 licenses available! and so they want everyone to compile scripts before submitting them as batch jobs to reduce license usage. mathworks sells licenses to universities at a discount, and then upcharges every other institution that might need them
this is related to licensing. it’s not just that matlab costs money but that it places heavy restrictions on what you can do with it. if you’re building packages that you want to distribute to other people, you need tests. if you use an open source language, there are a large number of free services, including Github, which will allow you to run your tests on their computers for free everytime you push a change to the code or open a pull request. this is super useful! you cannot do this with matlab, because their license doesn’t allow it. most researchers won’t need to do this, but neuro and psych research is built on a large collection of resaercher-created software: psychtoolbox, vista tools, freesurfer, nipype, fmriprep, etc and so this will affect researchers who use this softaware, which is everyone. similarly, mathworks has their own cloud that they want people to use, so it doesn’t work on google cloud, AWS, etc, whereas any open source language will
with that licensing comes control. mathworks decides what to do with matlab, and the interests of a for-profit company are never going to be the interests of us, as a community of academic researchers, so they’re going to make different choices about what to develop and prioritize. for one example, keeping track of dependencies is absolutely crucial for developing software in this federated manner, where I write something that uses pytorch, which uses numpy, etc. like I said, this happens in matlab too – we all use software written by researchers, not mathworks. but matlab doesn’t have an official declarative dependency manager, so tracking dependencies is incredibly difficult! this leads to things like people keeping copies of all their dependencies, having no idea which toolboxes or version numbers their code requires, etc. this has led to the development of three different unofficial dependency managers, including one by David Brainard’s lab, all of which were built by neuroscientists rather than experts. we shouldn’t need to do that! dependency management is really really hard and we should be able to trust it to experts. but it is unlikely that mathworks will do it, because they want everyone to see matlab as a thing that comes from them, they do not do a good job facilitating code sharing
and then finally, there’s the ideological point. we do science for the public good, and that means that the work we do should be available to others. like the for profit publishing companies, such as Elsevier and SpringerNature, for profit software companies actively work against the interests of science, for the reasons I just explained and makes money from doing so! so I just don’t think we should be supporting them

)