CCN Salon

Billy Broderick

August 11, 2023

Who am I?

PhD: NYU Center for Neural Science, May 2022
- Advisors: Eero Simoncelli and Jonathan Winawer
Research focus: how vision changes across the visual field, using fMRI, psychophysics, and computational models
Fell down the open source / open science rabbit hole

Many of you know who I am, my name is Billy, and I got my PhD from NYU’s Center for Neural Science, advised by Eero and Jon Winawer, who’s in the psychology department at NYU

I studied how vision changes across the visual field, using fMRI, psychophysics, and computational models. So if you’re staring at me, you have no problem making out my face, but if you look at the screen, my face loses a lot of detail – what detail is lost and where, that’s what I studied

and I really fell down the open source and open science rabbit hole over the course of my PhD. I was working as an RA doing fMRI research and applying for grad school in 2014/2015, which involved me trying to learn python and was also around the time when the reproducibility crisis in psychology really took off. so learning how to do reproducible research, how to share data, fundamentally how to do science in a cumulative way, so that we can actually build on each other’s work, has been very important to me.

and that means I became a bit of an open source partisan – my research stack is completely open source programs (hence why this is a reveal.js presentation, not powerpoint / keynote – happy to help spread the good word if anyone wants wants to uninstall their OS and go all Linux), I shared all the code and all the data I did during my PhD, and, most relevant to this talk, I started working on several python projects with other members of Eero’s lab, one of which became the project I’m talking about today

plen-1.0.1-intro-1.svg

plen-1.0.1-intro-2.svg

Simulate responses

Fit parameters

Synthesize stimuli

Metamers

Physically distinct stimuli that are perceptually identical

Metamers

Physically distinct stimuli that are perceptually identical

Metamers

Physically distinct stimuli that are perceptually identical (to a computational model)

plenoptic contains four synthesis methods, which have all been developed in Eero’s lab over the years. I won’t go though the rest of them in detail, but I’m happy to provide more detail or discuss them later if you’re interested

each of these methods and models was developed by members of the lab over the years, but generally for a specific research question and with an implementation that others could not use, at least not easily. the goal of plenoptic was to implement these methods in a general, model-agnostic way, taking advantage of pytorch’s automatic differentiation, so that we and other members of the scientific community could use them for novel models and questions

the package also contains several models and model components that we find useful and think others might. these are all implemented in pytorch, can be used easily with our synthesis methods, and are tested and maintained

that’s the over view of the contents, now let’s talk about the bigger picture questions

Why do this?

Improve understanding of computational models
Image space is vast!
Better understand single model or compare between competing models

For those of you who haven’t heard of this approach, you might wonder, why do this?

the goal is to improve our understanding of how our computational models make sense of their input. as adversarial examples and other similar techniques have shown, that’s not trivial! just because a model does well on image net doesn’t mean it won’t behave unexpectedly on new / out-of-distribution data

image space is vast, so any possible dataset is a tiny sample of it, and the methods in plenoptic provide a different and targeted way of exploring this space.

they can be used for:

improving understanding of a given model – for example, fit a model to a visual neuron, then generate some stimuli with plenoptic and use it in a new experiment
another way of comparing models: many models can do equivalently (or near equivalently) well on a given benchmark, and plenoptic’s methods provide a complementary approach for comparing them.

Status and Roadmap

All methods, models are implemented
- Currently improving geodesics
Working on improving tutorials and documentation, cleaning up code base
Looking for feedback and users!
- https://plenoptic.readthedocs.io/

Developers

Website

https://plenoptic.readthedocs.io/

How to write scientific software?

How do I make my package pip installable? (i.e., pip install my_awesome_package)
What’s the difference between requirements.txt, environment.yml, setup.py, and pyproject.toml and which should I use?
How do I get up on the Python package index (https://pypi.org)?
How do I make this work on other operating systems?
How do I automatically compile the C code my Python function uses?
How do I tell people how to use my code?
How do I get that information up on a website?
How do I make sure my code actually works?
How do I write code with other people?

these are the four main software packages we’re working on right now. we’re also thinking about what other packages to work on, including a manifold analysis package with SueYeon

Now to change directions a bit and talk about another project that the data scientists have been working on.

as you’ve heard, all of us data scientists are neuroscientists by training, who realized we like writing scientific software. but none of us are formally trained in software engineering and so, while we’ve all worked on python libraries before (that is, general purpose code we expect to be used by other people on a variety of machines), we had to figure everything out as we go.

and there’s a big gap between “I can write python code that works for me” and “I can write a python library that other people can install and use without any problems”. that gap is full of questions like:

how do i make my package pip installable? which leads to questions like: read through C code

which leads to other related questions like: …

and most importantly: how do I write code with other people?

we all figured out different ways to do this, but we’re all self-taught, and so we took the opportunity to come together, talk through how we’ve handled this in the past, and more importantly: put together a github repo and website that encodes this information.

it’s very much a work in progress, but I’m going to shift over and show you what we have:

docs website. show structure, then packaging, documentation, linters+tests
not done: show docstrings
switch to gh repo, look at mkdocs.yml, readthedocs.yml files

this is largely for us: so we make our decisions once, have thought through what to do and explained how/why to do it. it’s probably more indepth than most researchers want for their code, but in case anyone’s interested in these topics, it might be helpful to check out.

we’re planning on including a section that puts these in order, so researchers who want to share their code for others to you can see what are the steps in escalating “seriousness”: from the easy+necessary (adding a license and install instructions) to the harder / less necessary (building documentation on readthedocs, deploying to pip)

Events

Fall 2023:
- pynapple McGill workshop
- MIT Open Neurophysiology Symposium (DANDI, Miniscope, NWB, DataJoint, etc.)
- US Research Software Engineer Association Conference (1st annual!)
- Caiman workshop at Society for Neuroscience conference (for users)
Spring 2024:
- CaImAn code spring at Flatiron (for contributors)
- Open source systems neuroscience workshop? at Computational and Systems Neuroscience conference (Cosyne)
- plenoptic workshop? at Vision Science Society conference (VSS)
- pynapple + CaImAn workshop? at Federation of European Neuroscience Societies (FENS)
Summer 2024:
- Interns!

and then finally, we’re planning on being involved in a variety of events, both ones we’re hosting and taking part in

With these events, we’re looking to engage with several different communities:

users (McGill)
contributors (code sprint)
other neuroscience developers (Open Neurophysiology Symposium)
other research software engineers (RSE con)

some of these are more speculative than others, but we’re thinking about how to engage with these communities, and happy to hear suggestions for how we should reach and work with users and collaborators.

I’d also like to point out that we’ve been discussing having interns next summer to work on some of these projects as well, though we’re still working out exactly what that will look like

and that’s it, thank you for coming

CCN Salon

Billy Broderick

August 11, 2023

Who am I?

Simulate responses

Fit parameters

Synthesize stimuli

Metamers

Metamers

Metamers

Metamers

Metamers

Contents

Why do this?

Status and Roadmap

Developers

Developers

Website

How to write scientific software?

Events