William Broderick's Blog

snakemake, singularity, and HPC

2022-08-01T00:00:00-04:00

This is a bit of a sequel to my post last May about using conda environments with snakemake on the HPC. That solution worked for me, but was hacky and limited. This post will describe a different solution, which uses a singularity container containing a conda environment that can be used by snakemake for local use or use with SLURM, and has the ability to mount extra dependencies.

I put this together for my spatial frequency preferences project, and you can read that repo's README for more details on how to use it. The following post will attempt to describe the different components of it and why they work, which will hopefully make it easier to modify for other uses. The following description is all for NYU's greene cluster (which uses SLURM), but can hopefully be modified to other clusters and job submission systems.

My goal was to make it easier to rerun my analyses, allowing other people to make use of snakemake and the cluster to run them quickly, without requiring lots of setup and understanding of snakemake, SLURM, or singularity. To do that, I wanted to create a single container with a wrapper script that would be able to:

Run locally.
Run in an interactive session on greene.
Use snakemake to handle job submission on greene.
Mount paths for additional requirements that cannot be easily included in the container.
Be backed up for long-term archival.
Would not require the user to modify any source code themselves.

In order to do accomplish the first three goals, I needed to have the container and script work with both docker and singularity, as docker does not play nicely with computing clusters (since it wants to run everything with sudo permissions) and singularity is difficult for the typical user to set up on their personal machine.

The fourth goal was necessary because some of the requirements for re-running my full analysis have non open-source licenses (Matlab requires the user to pay for a license, and FSL and Freesurfer both require registration), which makes including them in a container difficult (with Matlab, at least, I could compile the required code into mex files and include those, but I've never done that before, I didn't write the required Matlab code (and so don't understand perfectly) and from preliminary research and discussion with NYU HPC staff, I don't think it's as straightforward as I had originally hoped).

The fifth goal is necessary because Docker hub makes no promises about its images staying around forever (and, in particular, is planning on deleting the images from free Docker accounts after six months of inactivity), so I can't rely on users always being able to find it there.

In order to accomplish the above, we need to create a container that contains all the dependencies we can and allows for mounting the ones we can't include. We will create a wrapper script, in python, which handles a lot of boilerplate, mounting the required paths and remapping arguments that get passed to snakemake so that it uses the correct paths for within the container. We also want this wrapper script to work with the default python 3 libraries, since any additional packages will only be available within the container. Finally, we need to configure snakemake so that it uses the container on jobs that it submits to the cluster.

With all that in mind, let's walk through the files that implement this solution. It involves four files from my spatial-frequency-preferences repo (build_docker, singularity_env.sh, run_singularity.py, and config.json), plus the singularity branch of my slurm snakemake profile.

`build_docker`:

# This is a Dockerfile, but we use a non-default name to prevent mybinder from
# using it. we also don't expect people to build the docker image themselves, so
# hopefully it shouldn't trip people up.

FROM mambaorg/micromamba

# git is necessary for one of the packages we install via pip, gcc is required
# to install another one of those packages, and we need to be root to use apt
USER root
RUN apt -y update
RUN apt -y install git gcc

# switch back to the default user
USER micromamba

# create the directory we'll put our dependencies in.
RUN mkdir -p /home/sfp_user/
# copy over the conda environment yml file
COPY ./environment.yml /home/sfp_user/sfp-environment.yml

# install the required python packages and remove unnecessary files.
RUN micromamba install -n base -y -f /home/sfp_user/sfp-environment.yml && \
    micromamba clean --all --yes

# get the specific commit of the MRI_tools repo that we need
RUN git clone https://github.com/WinawerLab/MRI_tools.git /home/sfp_user/MRI_tools; cd /home/sfp_user/MRI_tools; git checkout 8508652bd9e6b5d843d70be0910da413bbee432e
# get the matlab toolboxes that we need
RUN git clone https://github.com/cvnlab/GLMdenoise.git /home/sfp_user/GLMdenoise
RUN git clone https://github.com/vistalab/vistasoft.git /home/sfp_user/vistasoft

# get the slurm snakemake profile, the singularity branch
RUN mkdir -p /home/sfp_user/.config/snakemake
RUN git clone -b singularity https://github.com/billbrod/snakemake-slurm.git /home/sfp_user/.config/snakemake/slurm
# copy over the env.sh file, which sets environmental variables
COPY ./singularity_env.sh /home/sfp_user/singularity_env.sh

This is a standard Dockerfile (we use a different name because we also want to be able to launch a Binder instance from this repo, and Binder defaults to using Dockerfile if present), which builds on the mamba container (mamba is a drop-in replacement for conda that, in my experience, solves the environment much quicker). The main thing it does is install the conda environment used by my analysis. It also grabs the other git repos that are required: the MRI_tools repo (used for pre-processing the fMRI data), two matlab packages, and the singularity branch of my slurm snakemake profile.

To then get this image onto the cluster, you'll also need to push it to DockerHub, so build and push it:

sudo docker build --tag=billbrod/sfp:latest -f build_docker  ./
sudo docker push  billbrod/sfp:latest

`singularity_env.sh`

#!/usr/bin/env bash

# set up environment variables for other libraries, add them to path
export FREESURFER_HOME=/home/sfp_user/freesurfer
export PATH=$FREESURFER_HOME/bin:$PATH

export PATH=/home/sfp_user/matlab/bin:$PATH

export FSLOUTPUTTYPE=NIFTI_GZ
export FSLDIR=/home/sfp_user/fsl
export PATH=$FSLDIR/bin:$PATH

# modify the config.json file so it points to the location of MRI_tools,
# GLMDenoise, and Vistasoft within the container
if [ -f /home/sfp_user/spatial-frequency-preferences/config.json ]; then
    cp /home/sfp_user/spatial-frequency-preferences/config.json /home/sfp_user/sfp_config.json
    sed -i 's|"MRI_TOOLS":.*|"MRI_TOOLS": "/home/sfp_user/MRI_tools",|g' /home/sfp_user/sfp_config.json
    sed -i 's|"GLMDENOISE_PATH":.*|"GLMDENOISE_PATH": "/home/sfp_user/GLMdenoise",|g' /home/sfp_user/sfp_config.json
    sed -i 's|"VISTASOFT_PATH":.*|"VISTASOFT_PATH": "/home/sfp_user/vistasoft",|g' /home/sfp_user/sfp_config.json
fi

This file gets copied into the container and will get sourced as soon as the container is started up (see the run_singularity.py section below for how we do this). It sets up environmental variables for the extra dependencies and adds them to path, as well as modifying the config.json path to point where those packages are located within the container. Note that these software packages (matlab, FSL, and Freesurfer) are not included in the container, but because of how we've set up the run_singularity.py script, we know where they'll be mounted.

`config.json`

{
  "DATA_DIR": "/scratch/wfb229/sfp",
  "WORKING_DIR": "/scratch/wfb229/preprocess",
  "MATLAB_PATH": "/share/apps/matlab/2020b",
  "FREESURFER_HOME": "/share/apps/freesurfer/6.0.0",
  "FSLDIR": "/share/apps/fsl/5.0.10",
  "MRI_TOOLS": "/home/billbrod/Documents/MRI_tools",
  "GLMDENOISE_PATH": "/home/billbrod/Documents/MATLAB/toolboxes/GLMdenoise",
  "VISTASOFT_PATH": "/home/billbrod/Documents/MATLAB/toolboxes/vistasoft",
  "TESLA_DIR": "/mnt/Tesla/spatial_frequency_preferences",
  "EXTRA_FILES_DIR": "/mnt/winawerlab/Projects/spatial_frequency_preferences/extra_files",
  "SUBJECTS_DIR": "/mnt/winawerlab/Freesurfer_subjects",
  "RETINOTOPY_DIR": "/mnt/winawerlab/Projects/Retinotopy/BIDS"
}

Snakemake allows for a configuration file, either yml or json, which we use to specify a variety of paths. We use json here, even though it doesn't allow for comments, because it can be parsed by the standard python library. These paths should all be set to locations on your machine / the cluster (not within the container). The above is an example that works for my user on the NYU greene cluster.

When using the container, only the first five paths need to be set (from DATA_DIR to FSLDIR; the final ones are used either when running without the container or when copying data into a BIDS-compliant format). DATA_DIR gives the location of the data set and where we'll place the output of the analysis and WORKING_DIR is a working directory for preprocessing and is only used temporarily in that step. The next three are the root directory of the installations for matlab, Freesurfer, and FSL: to find their locations, make sure they're on your path (if you're on a cluster, this is probably by using module load) and then run e.g., which matlab (or which mri_convert, etc.) to find where they're installed. Note that we want the root directory of the install (not the bin/ folder containing the binary executables so that if which matlab returns /share/apps/matlab/2020b/bin/matlab, we just want /share/apps/matlab/2020b).

Of all the files needed for this process, this is the only one that requires modification by the user, and my spatial-frequency-preferences readme includes a long description of what the different fields are, which need to be set, etc.

`run_singularity.py`

This is the main script that the user will run, which I generally refer to as the "wrapper script". If everything is working, the user will simply pass the command they wish to run to this script and it will bind the various directories, set environmental variables, and properly construct the arguments for singularity or docker, whichever the user wishes to use.

Because this script is so much larger than the rest, we'll step through it section by section, and I'll include the full script at the end.

#!/usr/bin/env python3

import argparse
import subprocess
import os
import os.path as op
import json
import re
from glob import glob

First, we specify this script must be run with python 3 and import the necessary python modules.

# slurm-related paths. change these if your slurm is set up differently or you
# use a different job submission system. see docs
# https://sylabs.io/guides/3.7/user-guide/appendix.html#singularity-s-environment-variables
# for full description of each of these environmental variables
os.environ['SINGULARITY_BINDPATH'] = os.environ.get('SINGULARITY_BINDPATH', '') + ',/opt/slurm,/usr/lib64/libmunge.so.2.0.0,/usr/lib64/libmunge.so.2,/var/run/munge,/etc/passwd'
os.environ['SINGULARITYENV_PREPEND_PATH'] = os.environ.get('SINGULARITYENV_PREPEND_PATH', '') + ':/opt/slurm/bin'
os.environ['SINGULARITY_CONTAINLIBS'] = os.environ.get('SINGULARITY_CONTAINLIBS', '') + ',' + ','.join(glob('/opt/slurm/lib64/libpmi*'))

The next several lines handle some slurm-related boilerplate. We take some singularity-related environmental variables and add on paths related to the slurm configuration so that we'll have access to the slurm commands (e.g., sbatch) from within the container. These will likely vary across SLURM clusters and so will need to be modified if using this on any cluster other than NYU's greene.

def check_singularity_envvars():
    """Make sure SINGULARITY_BINDPATH, SINGULARITY_PREPEND_PATH, and SINGULARITY_CONTAINLIBS only contain existing paths
    """
    for env in ['SINGULARITY_BINDPATH', 'SINGULARITYENV_PREPEND_PATH', 'SINGULARITY_CONTAINLIBS']:
        paths = os.environ[env]
        joiner = ',' if env != "SINGULARITYENV_PREPEND_PATH" else ':'
        paths = [p for p in paths.split(joiner) if op.exists(p)]
        os.environ[env] = joiner.join(paths)


def check_bind_paths(volumes):
    """Check that paths we want to bind exist, return only those that do."""
    return [vol for vol in volumes if op.exists(vol.split(':')[0])]

The next two functions make sure we only pass through existing paths to the container, either via the environmental variables discussed above, or via the user-specified bind paths. By excluding non-existing directories from the bind paths, we allow the user to ignore any options they don't want to set, so that they can ignore mounting any directories containing software they don't need.

def main(image, args=[], software='singularity', sudo=False):
    """Run sfp singularity container!

    Parameters
    ----------
    image : str
        If running with singularity, the path to the .sif file containing the
        singularity image. If running with docker, name of the docker image.
    args : list, optional
        command to pass to the container. If empty (default), we open up an
        interactive session.
    software : {'singularity', 'docker'}, optional
        Whether to run image with singularity or docker
    sudo : bool, optional
        If True, we run docker with `sudo`. If software=='singularity', we
        ignore this.

    """

The main() function accepts the same arguments as the command-line script, which we'll discuss at the end.

check_singularity_envvars()
with open(op.join(op.dirname(op.realpath(__file__)), 'config.json')) as f:
    config = json.load(f)
volumes = [
    f'{op.dirname(op.realpath(__file__))}:/home/sfp_user/spatial-frequency-preferences',
    f'{config["MATLAB_PATH"]}:/home/sfp_user/matlab',
    f'{config["FREESURFER_HOME"]}:/home/sfp_user/freesurfer',
    f'{config["FSLDIR"]}:/home/sfp_user/fsl',
    f'{config["DATA_DIR"]}:{config["DATA_DIR"]}',
    f'{config["WORKING_DIR"]}:{config["WORKING_DIR"]}'
]
volumes = check_bind_paths(volumes)
# join puts --bind between each of the volumes, we also need it in the
# beginning
volumes = '--bind ' + " --bind ".join(volumes)

First, main() takes care of the paths and environmental variables. We check the singularity-related environmental variables, as discussed above, then load in the config.json file, also discussed earlier. We use the user-supplied values here to set up the arguments for the --bind flag (equivalent to docker's --volume flag). Note that the software-related paths (the ones containing the project repo, matlab, freesurfer, and FSL) are all remapped to static paths within /home/sfp_user/, while the data-related paths (DATA_DIR and WORKING_DIR) are left unchanged. The data paths are unchanged because there are too many references to them in how snakemake structures the commands for me to comprehensively remap, so we just preserve them.

We then check that all the paths that we'll be binding exist and remove any that don't with the check_bind_paths() function, discussed above. Finally, we combine that list of volumes into a string to pass through to singularity, which will be formatted like --bind VOL1 --bind VOL2 ....

# if the user is passing a snakemake command, need to pass
# --configfile /home/sfp_user/sfp_config.json, since we modify the config
# file when we source singularity_env.sh
if args and 'snakemake' == args[0]:
    args = ['snakemake', '--configfile', '/home/sfp_user/sfp_config.json',
            '-d', '/home/sfp_user/spatial-frequency-preferences',
            '-s', '/home/sfp_user/spatial-frequency-preferences/Snakefile', *args[1:]]

args specifies what the user wants to do with the container, and there are four possibilities:

args is empty, in which case we open up an interactive session.
args is a list of strings, and the first string is snakemake.
args is a single string, and starts with snakemake.
args is either a single string or a list of strings, and doesn't contain snakemake.

The above section handles possibility \#2. In this case, we know that the arguments contains no flags for snakemake (in that case, args would have to be quoted and thus we'd be in possibility \#3). We therefore modify the command to be run from inside the container, adding the specification of the config file, which we modified with singularity_env.sh, and specifying the paths to both the working directory and the Snakefile (these last two mean that we can run the snakemake command from other paths within the container). Finally, we append the rest of the user-specified arguments.

# in this case they passed a string so args[0] contains snakemake and then
# a bunch of other stuff
elif args and args[0].startswith('snakemake'):
    args = ['snakemake', '--configfile', '/home/sfp_user/sfp_config.json',
            '-d', '/home/sfp_user/spatial-frequency-preferences',
            '-s', '/home/sfp_user/spatial-frequency-preferences/Snakefile', args[0].replace('snakemake ', ''), *args[1:]]
    # if the user specifies --profile slurm, replace it with the
    # appropriate path. We know it will be in the last one of args and
    # nested below the above elif because if they specified --profile then
    # the whole thing had to be wrapped in quotes, which would lead to this
    # case.
    if '--profile slurm' in args[-1]:
        args[-1] = args[-1].replace('--profile slurm',
                                    '--profile /home/sfp_user/.config/snakemake/slurm')
    # then need to make sure to mount this
    elif '--profile' in args[-1]:
        profile_path = re.findall('--profile (.*?) ', args[-1])[0]
        profile_name = op.split(profile_path)[-1]
        volumes.append(f'{profile_path}:/home/sfp_user/.config/snakemake/{profile_name}')
        args[-1] = args[-1].replace(f'--profile {profile_path}',
                                    f'--profile /home/sfp_user/.config/snakemake/{profile_name}')

This block corresponds to possibility \#3. The beginning is very similar to \#2 explained above, except that we include the rest of args[0], after having removed the word snakemake.

The following sections either remap the snakemake profile from slurm to the slurm profile within the container, or find the specified path of the snakemake profile and add it to the list of volumes to mount within the container. Note that this will fail if the user just specifies the name of the profile, rather than giving the full path. This could be made more general by checking whether the profile_path variable grabbed with regex looks like a path, and, if not, searching the locations that snakemake searches for profiles (notably, ~/.config/snakemake), but that seemed like a lot of work.

# open up an interactive session if the user hasn't specified an argument,
# otherwise pass the argument to bash. regardless, make sure we source the
# env.sh file
if not args:
    args = ['/bin/bash', '--init-file', '/home/sfp_user/singularity_env.sh']

This block corresponds to situation \#1: no arguments passed. In this case, we simply open up an interactive bash session, sourcing singularity_env.sh before we start.

else:
    args = ['/bin/bash', '-c',
            # this needs to be done with single quotes on the inside so
            # that's what bash sees, otherwise we run into
            # https://stackoverflow.com/questions/45577411/export-variable-within-bin-bash-c;
            # double-quoted commands get evaluated in the *current* shell,
            # not by /bin/bash -c
            f"'source /home/sfp_user/singularity_env.sh; {' '.join(args)}'"]

This block handles situations 2 through 4: we pass the arguments through to the bash interpreter, making sure to source singularity_env.sh first. In an ideal world, this sourcing could be done with the same --init-file arg as used in the previous block, but that wasn't working for me. Note the importance of single quotes.

# set these environmental variables, which we use for the jobs submitted to
# the cluster so they know where to find the container and this script
env_str = f"--env SFP_PATH={op.dirname(op.realpath(__file__))} --env SINGULARITY_CONTAINER_PATH={image}"

These two environmental variables are used by our snakemake slurm profile, described in the next section, so that all our submitted jobs know where the container and the wrapper script are found, since they'll be used by them as well.

# the -e flag makes sure we don't pass through any environment variables
# from the calling shell, while --writable-tmpfs enables us to write to the
# container's filesystem (necessary because singularity_env.sh makes a
# temporary config.json file)
if software == 'singularity':
    exec_str = f'singularity exec -e {env_str} --writable-tmpfs {volumes} {image} {" ".join(args)}'

Now, we combine all the strings we've been configuring into a single string that we can execute. The only new components are the -e flag, which ensures we don't send through any extra environmental variables from the calling shell, and the --writable-tmpfs flag, which allows us to write to the container's filesystem (not just the mounted ones), which we need because we modify the snakemake configuration file.

elif software == 'docker':
    volumes = volumes.replace('--bind', '--volume')
    exec_str = f'docker run {volumes} -it {image} {" ".join(args)}'
    if sudo:
        exec_str = 'sudo ' + exec_str

If we're using docker instead of singularity, the command is slightly different: we replace --bind with --volume and change some of the other flags. This command doesn't use the -e flag and the extra environmental variable manipulations that singularity requires, because docker cannot be used on the cluster, and so this will not need to interact with the job scheduler (e.g., SLURM).

print(exec_str)
# we use shell=True because we want to carefully control the quotes used
subprocess.call(exec_str, shell=True)

Finally, we print out the command we're running (largely for debugging purposes, I'm not sure if this would be that useful to the user), and call it using subprocess.

if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description=("Run billbrod/sfp container. This is a wrapper, which binds the appropriate"
                     " paths and sources singularity_env.sh, setting up some environmental variables.")
    )
    parser.add_argument('image',
                        help=('If running with singularity, the path to the '
                              '.sif file containing the singularity image. '
                              'If running with docker, name of the docker image.'))
    parser.add_argument('--software', default='singularity', choices=['singularity', 'docker'],
                        help="Whether to run this with singularity or docker")
    parser.add_argument('--sudo', '-s', action='store_true',
                        help="Whether to run docker with sudo or not. Ignored if software==singularity")
    parser.add_argument("args", nargs='*',
                        help="Command to pass to the container. If empty, we open up an interactive session.")
    args = vars(parser.parse_args())
    main(**args)

The very bottom of the script contains the argparse configuration, which makes the help string from the command line more informative.

The users calls the script like so: ./run_singularity.py path/to/singularity_image.sif CMD if using singularity, and ./run_singularity.py user/image_name --software docker -s CMD if using docker (that -s flag runs docker as sudo, and so may not be necessary, depending on their docker configuration). Note that this CMD is the equivalent of args discussed above. CMD can be left empty (in which case an interactive session will be opened) or a string that will be run inside the container, such as the snakemake commands required to recreate the analysis, e.g., 'snakemake main_figure_paper' (for my spatial-frequency-preferences project). Note that single quotes are necessary if any flags are included in CMD, in order to prevent run_singularity.py from trying to interpret them.

Finally, the complete script:

#!/usr/bin/env python3

import argparse
import subprocess
import os
import os.path as op
import json
import re
from glob import glob

# slurm-related paths. change these if your slurm is set up differently or you
# use a different job submission system. see docs
# https://sylabs.io/guides/3.7/user-guide/appendix.html#singularity-s-environment-variables
# for full description of each of these environmental variables
os.environ['SINGULARITY_BINDPATH'] = os.environ.get('SINGULARITY_BINDPATH', '') + ',/opt/slurm,/usr/lib64/libmunge.so.2.0.0,/usr/lib64/libmunge.so.2,/var/run/munge,/etc/passwd'
os.environ['SINGULARITYENV_PREPEND_PATH'] = os.environ.get('SINGULARITYENV_PREPEND_PATH', '') + ':/opt/slurm/bin'
os.environ['SINGULARITY_CONTAINLIBS'] = os.environ.get('SINGULARITY_CONTAINLIBS', '') + ',' + ','.join(glob('/opt/slurm/lib64/libpmi*'))


def check_singularity_envvars():
    """Make sure SINGULARITY_BINDPATH, SINGULARITY_PREPEND_PATH, and SINGULARITY_CONTAINLIBS only contain existing paths
    """
    for env in ['SINGULARITY_BINDPATH', 'SINGULARITYENV_PREPEND_PATH', 'SINGULARITY_CONTAINLIBS']:
        paths = os.environ[env]
        joiner = ',' if env != "SINGULARITYENV_PREPEND_PATH" else ':'
        paths = [p for p in paths.split(joiner) if op.exists(p)]
        os.environ[env] = joiner.join(paths)


def check_bind_paths(volumes):
    """Check that paths we want to bind exist, return only those that do."""
    return [vol for vol in volumes if op.exists(vol.split(':')[0])]


def main(image, args=[], software='singularity', sudo=False):
    """Run sfp singularity container!

    Parameters
    ----------
    image : str
        If running with singularity, the path to the .sif file containing the
        singularity image. If running with docker, name of the docker image.
    args : list, optional
        command to pass to the container. If empty (default), we open up an
        interactive session.
    software : {'singularity', 'docker'}, optional
        Whether to run image with singularity or docker
    sudo : bool, optional
        If True, we run docker with `sudo`. If software=='singularity', we
        ignore this.

    """
    check_singularity_envvars()
    with open(op.join(op.dirname(op.realpath(__file__)), 'config.json')) as f:
        config = json.load(f)
    volumes = [
        f'{op.dirname(op.realpath(__file__))}:/home/sfp_user/spatial-frequency-preferences',
        f'{config["MATLAB_PATH"]}:/home/sfp_user/matlab',
        f'{config["FREESURFER_HOME"]}:/home/sfp_user/freesurfer',
        f'{config["FSLDIR"]}:/home/sfp_user/fsl',
        f'{config["DATA_DIR"]}:{config["DATA_DIR"]}',
        f'{config["WORKING_DIR"]}:{config["WORKING_DIR"]}'
    ]
    volumes = check_bind_paths(volumes)
    # join puts --bind between each of the volumes, we also need it in the
    # beginning
    volumes = '--bind ' + " --bind ".join(volumes)
    # if the user is passing a snakemake command, need to pass
    # --configfile /home/sfp_user/sfp_config.json, since we modify the config
    # file when we source singularity_env.sh
    if args and 'snakemake' == args[0]:
        args = ['snakemake', '--configfile', '/home/sfp_user/sfp_config.json',
                '-d', '/home/sfp_user/spatial-frequency-preferences',
                '-s', '/home/sfp_user/spatial-frequency-preferences/Snakefile', *args[1:]]
    # in this case they passed a string so args[0] contains snakemake and then
    # a bunch of other stuff
    elif args and args[0].startswith('snakemake'):
        args = ['snakemake', '--configfile', '/home/sfp_user/sfp_config.json',
                '-d', '/home/sfp_user/spatial-frequency-preferences',
                '-s', '/home/sfp_user/spatial-frequency-preferences/Snakefile', args[0].replace('snakemake ', ''), *args[1:]]
        # if the user specifies --profile slurm, replace it with the
        # appropriate path. We know it will be in the last one of args and
        # nested below the above elif because if they specified --profile then
        # the whole thing had to be wrapped in quotes, which would lead to this
        # case.
        if '--profile slurm' in args[-1]:
            args[-1] = args[-1].replace('--profile slurm',
                                        '--profile /home/sfp_user/.config/snakemake/slurm')
        # then need to make sure to mount this
        elif '--profile' in args[-1]:
            profile_path = re.findall('--profile (.*?) ', args[-1])[0]
            profile_name = op.split(profile_path)[-1]
            volumes.append(f'{profile_path}:/home/sfp_user/.config/snakemake/{profile_name}')
            args[-1] = args[-1].replace(f'--profile {profile_path}',
                                        f'--profile /home/sfp_user/.config/snakemake/{profile_name}')
    # open up an interactive session if the user hasn't specified an argument,
    # otherwise pass the argument to bash. regardless, make sure we source the
    # env.sh file
    if not args:
        args = ['/bin/bash', '--init-file', '/home/sfp_user/singularity_env.sh']
    else:
        args = ['/bin/bash', '-c',
                # this needs to be done with single quotes on the inside so
                # that's what bash sees, otherwise we run into
                # https://stackoverflow.com/questions/45577411/export-variable-within-bin-bash-c;
                # double-quoted commands get evaluated in the *current* shell,
                # not by /bin/bash -c
                f"'source /home/sfp_user/singularity_env.sh; {' '.join(args)}'"]
    # set these environmental variables, which we use for the jobs submitted to
    # the cluster so they know where to find the container and this script
    env_str = f"--env SFP_PATH={op.dirname(op.realpath(__file__))} --env SINGULARITY_CONTAINER_PATH={image}"
    # the -e flag makes sure we don't pass through any environment variables
    # from the calling shell, while --writable-tmpfs enables us to write to the
    # container's filesystem (necessary because singularity_env.sh makes a
    # temporary config.json file)
    if software == 'singularity':
        exec_str = f'singularity exec -e {env_str} --writable-tmpfs {volumes} {image} {" ".join(args)}'
    elif software == 'docker':
        volumes = volumes.replace('--bind', '--volume')
        exec_str = f'docker run {volumes} -it {image} {" ".join(args)}'
        if sudo:
            exec_str = 'sudo ' + exec_str
    print(exec_str)
    # we use shell=True because we want to carefully control the quotes used
    subprocess.call(exec_str, shell=True)


if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description=("Run billbrod/sfp container. This is a wrapper, which binds the appropriate"
                     " paths and sources singularity_env.sh, setting up some environmental variables.")
    )
    parser.add_argument('image',
                        help=('If running with singularity, the path to the '
                              '.sif file containing the singularity image. '
                              'If running with docker, name of the docker image.'))
    parser.add_argument('--software', default='singularity', choices=['singularity', 'docker'],
                        help="Whether to run this with singularity or docker")
    parser.add_argument('--sudo', '-s', action='store_true',
                        help="Whether to run docker with sudo or not. Ignored if software==singularity")
    parser.add_argument("args", nargs='*',
                        help=("Command to pass to the container. If empty, we open up an interactive session."
                              " If it contains flags, surround with SINGLE QUOTES (not double)."))
    args = vars(parser.parse_args())
    main(**args)

slurm snakemake profile

If we want to submit jobs to the job scheduler, we need to tell snakemake how to do so and to use the container when it does so. We do this via a custom profile, which can be found as the singularity branch of my snakemake-slurm github repository. This is based on the canonical slurm snakemake profile. I originally created my version 5 years ago, and it's possible there are changes to the original in that time that would be helpful, so you can use mine or the original for this (the only changes I've made besides those described below are a small change to add support for the –gres option, required for requesting GPUs, and a small change to partition).

The only file that is specific to the implementation discussed in this post is slurm-jobscript.sh:

#!/bin/bash
#SBATCH --export=SINGULARITY_CONTAINER_PATH,SFP_PATH
# properties = {properties}

# q is a special formatting symbol used by snakemake to tell it to escape quotes
# correctly
$SFP_PATH/run_singularity.py $SINGULARITY_CONTAINER_PATH {exec_job:q}

Note that this script makes use of the SINGULARITY_CONTAINER_PATH and SFP_PATH environmental variables, which we made sure to set in the wrapper script run_singularity.py above. They specify the location of the singularity container containing the environment and the directory containing the code for the project (and thus, run_singularity.py).

Each job that snakemake submits to the cluster will use this script as the template for its job, and so we are ensuring that each of those jobs will use run_singularity.py to run within the container. We also use the :q formatting symbol (which is a special snakemake one, not available in standard python) to escape quotes correctly, which is important since, as discussed in the previous section, we need to control the single quotes carefully to make sure the flags are interpreted by the right software.

Archiving

Finally, we would like to be able to back up the containers for long-term archiving. I wrote this code as a way of making my analysis and figure-creation reproducible for a scientific paper. This means that I would like the code to remain runnable for as long as possible and it is unlikely that many people will use it. Docker hub had planned to delete images from free Docker accounts after six months of activity, though it's unclear what the status of that plan is. Regardless, we should not be relying on Docker hub for long-term archiving and so we need another solution.

Fortunately, singularity creates a .sif file containing the container when pulling it, and docker can export the container into a .tar file using docker save (e.g., docker save billbrod/sfp:v1.0.0 > sfp_v1.0.0_docker.tar). These files can then be archived like any other file. They are rather large (2.5GB for the .sif file, 5.1GB for the .tar file), and so something like the Open Science Framework, where I normally place my academic research-related files, is not an option.

Ultimately, I used NYU's Faculty Digital Archive. Something like Amazon Web Service's S3 buckets (or other cloud storage) would be another option, though not free. When looking for where to archive, any service you use should give objects a doi and allow for versioning. Ideally, it should be run by a non-profit and open source, but that might not be possible. If you're at a university, I recommend reaching out to your university library or HPC team to see if they have any suggestions. Fundamentally, archiving and sharing the container is not that different from archiving and sharing a large data set.

Archiving the container somewhere publicly-accessible allows users to download the container and use it, even if the docker hub image is deleted. With singularity, the container is used directly, whereas with docker, docker load is required (e.g., docker load < sfp_v1.0.0_docker.tar).

Conclusion

This post outlines how to create and use a container which contains all dependencies possible, allows for mounting any additional dependencies (such as those that require a license for use), and can be used identically locally and on the cluster, as well as being able to use snakemake to submit jobs on a cluster. While the scripts I wrote are all slurm-specific (and, even more so, specific to NYU's greene cluster), hopefully they can provide a jumping off point for others.

Scripting figure creation with inkscape and python

2021-11-06T00:00:00-04:00

I spent some time figuring out how to use inkscape and python to script the creation of figures that contain bitmaps, allowing for the user to specify the DPI when embedded. Normally, the way to handle this involves creating the figures directly in a vector graphics editor like inkscape or Adobe Illustrator, which works fine, but I like to script as much as of my workflow as possible, to make testing different figure versions as quick as possible. I spent probably too much time getting it to work, but here's an example GitHub repo I made showing how the process works, with some tips on how to use it yourself.

conda, snakemake, and HPC

2021-05-05T00:00:00-04:00

NOTE: I have since come up with a solution that I like better / think is less hacky, described in this post.

NYU's new high performance computing (HPC) cluster, Greene, was put into production earlier this year and, on the new system, they want all users to use singularity containers for conda environments. This is because conda environments create a bunch of files: one conda environment can contain more than 100 thousand files; on Greene, each user is limited to 1 million files, so it's not great to have that many be eaten up by conda. To get around this, they want everyone to use overlay singularity containers, and have put together some documentation to help users get set up.

However, I use the workflow management system snakemake to run my analyses and found it difficult to set up my environments such that snakemake could make use of them. After meeting with NYU HPC staff and some extra work on both their part and mine, I think I've come up with a solution. I'm not sure if this will help anyone outside of NYU, but I figured it was worth writing up.

As detailed on the HPC site, first pick an appropriate overlay filesystem (based on the number of files and overall size; the one I pick probably has more files than necessar for a single conda environment, and you would need more space if you wanted to install large packages like the recent version of pytorch) and copy it to a new directory in your /scratch/ directory (we'll be calling that directory overlay/ throughout):

``` bash mkdir /scratch/$USER/overlay cd /scratch/$USER/overlay

if you're not on NYU's Greene cluster, this path will almost certainly be

different

cp /scratch/work/public/overlay-fs-ext3/overlay-5GB-3.2M.ext3.gz . gunzip overlay-5GB-3.2M.ext3.gz ```
Run a container with the overlay filesystem. You'll have to pick what flavor of linux you want, but it shouldn't matter for our purposes (I picked Ubuntu because it's what I'm most familiar with; CentOS is the version used on most HPC systems, including Greene, so you may want to use that):

``` bash

again, if you're not on Greene, the path to the linux singularity image will

almost certainly be different

singularity exec --overlay /scratch/$USER/overlay/overlay-5GB-3.2M.ext3 /scratch/work/public/singularity/ubuntu-20.04.1.sif /bin/bash ```
You are now inside the singularity container. We're going to install our environment in /ext3 using miniconda. Note that you can skip this and the following step on greene by using an HPC-provided script; instead, run bash /setup/apps/utils/singularity-conda/setup-conda.bash.

bash cd /ext3 wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh sh Miniconda3-latest-Linux-x86_64.sh -b -p /ext3/miniconda3
Copy the following into the file /ext3/env.sh:

``` bash

!/bin/bash

this needs to be copied into the overlay, as /ext3/env.sh

source /ext3/miniconda3/etc/profile.d/conda.sh export PATH=/ext3/miniconda3/bin:$PATH ```
/ext3/env.sh is also where you'd modify the path or set environmental variables for additional packages; lmod is not available in the singularity container, so you can't use module load and will have to configure your paths yourself. To see what module load would do, you can run module show freesurfer/6.0.0 (for example) instead, which will show you how the path is modified and what environmental variables are set. For example, for an fMRI project I've been working on, I need access to FSL, Freesurfer, and MATLAB, so I add the following to the end of my env.sh file:

``` bash

set up other libraries (module doesn't work in the container)

export FREESURFER_HOME=/share/apps/freesurfer/6.0.0 export PATH=$FREESURFER_HOME/bin:$PATH export SUBJECTS_DIR=/scratch/wfb229/sfp_minimal/derivatives/freesurfer

export PATH=/share/apps/matlab/2020b/bin:$PATH

export FSLOUTPUTTYPE=NIFTI_GZ export FSLDIR=/share/apps/fsl/5.0.10 export PATH=$FSLDIR/bin:$PATH ```
Run source /ext3/env.sh to source the above file, configuring your path so conda is found on it. Run which conda and which python to make sure: it should show /ext3/miniconda3/bin/conda and /ext3/miniconda3/bin/python, respectively.
Install your conda environment like normal, using conda install and pip install. (Note: this post assumes you're installing your environment in the miniconda base environment, since we'll only have a single environment per overlay container; you could probably install it in a new environment as well, as long as you modify the following /ext3/env.sh so that it ends with conda activate <my-env>). For example, let's install numpy and snakemake:

bash conda install numpy snakemake -c bioconda
Exit out of the singularity container. We now need to allow snakemake to access this environment. For that, two components are required: a snakemake profile for slurm and a way to redirect the python and snakemake commands so they use the versions found within the container. Download the snakemake-slurm repo and place it in ~/.config (the important part of this config for the purposes of this post is slurm-jobscript.sh, particularly the lines where we modify the path):

``` bash

exit the container

exit

create the snakemake slurm profile

mkdir -p ~/.config/snakemake cd ~/.config/snakemake github clone git@github.com:billbrod/snakemake-slurm.git slurm ```
Copy the following into the file /scratch/$USER/overlay/python.sh:

``` bash

!/bin/bash

https://stackoverflow.com/questions/1668649/how-to-keep-quotes-in-bash-arguments

args='' for i in "$@"; do i="${i//\/\\}" args="$args \"${i//\"/\\"}\"" done

module purge

export PATH=/share/apps/singularity/bin:$PATH

file systems

export SINGULARITY_BINDPATH=/mnt,/scratch,/share/apps if [ -d /state/partition1 ]; then export SINGULARITY_BINDPATH=$SINGULARITY_BINDPATH,/state/partition1 fi

SLURM related

export SINGULARITY_BINDPATH=$SINGULARITY_BINDPATH,/opt/slurm,/usr/lib64/libmunge.so.2.0.0,/usr/lib64/libmunge.so.2,/var/run/munge,/etc/passwd export SINGULARITYENV_PREPEND_PATH=/opt/slurm/bin if [ -d /opt/slurm/lib64 ]; then export SINGULARITY_CONTAINLIBS=$(echo /opt/slurm/lib64/libpmi* | xargs | sed -e 's/ /,/g') fi

nv="" if [[ "$(hostname -s)" =~ ^g ]]; then nv="--nv"; fi cmd=$(basename $0)

singularity exec $nv \ --overlay /scratch/$USER/overlay/overlay-5GB-3.2M.ext3:ro \ /scratch/work/public/singularity/ubuntu-20.04.1.sif \ /bin/bash -c " source /ext3/env.sh $cmd $args exit "

```
Create symlinks for python, python3, and snakemake, all redirecting to our newly created python.sh:

bash cd /scratch/$USER/overlay ln -sv python.sh python ln -sv python.sh python3 ln -sv python.sh snakemake
Add the following lines to your .bashrc so that these symlinks are on your path. Exit and enter your shell so this modification takes effect. You can check this worked with which snakemake or which python, which should give you /scratch/$USER/overlay/snakemake and /scratch/$USER/overlay/python, respectively. Now, snakemake and python will both use that python.sh script, which runs the command using the singularity overlay image.

``` bash if [ "$SINGULARITY_CONTAINER" == "" ]; then export PATH=/scratch/$USER/overlay:$PATH fi

```
Now, this is the hacky part: start the overlay container back up, and modify the snakemake executor so it uses python instead of sys.executable (sys.executable will be the absolute path to a python interpreter and thus not use the sneaky symlinks we just created; bare python will use them because of how we've set up our path). Open up the singularity executor file; the exact path to this will depend on where you installed miniconda and your snakemake version, but on mine (python 3.7.8 and snakemake 5.4.5) it's /ext3/miniconda3/python3.7/site-packages/snakemake/executors.py (on more recent versions of snakemake, it will be snakemake/executors/__init__.py). Then find the lines where self.exec_job is being defined and {sys.executable} is used (lines 240 and 430 for my install) and replace {sys.executable} with python. Here's my diff as an example:

``` diff 240,241c240 < # '{sys.executable} -m snakemake {target} --snakefile {snakefile} ', < 'python -m snakemake {target} --snakefile {snakefile} ',
```
        '{sys.executable} -m snakemake {target} --snakefile {snakefile} ',
```
430,431c429 < # '{sys.executable} ' if assume_shared_fs else 'python ', < 'python ',
```
            '{sys.executable} ' if assume_shared_fs else 'python ',
```
```

That should work. I'm not super happy with having to modify snakemake's source code to get this working, but it does work. Let me know if you know of a better solution!

Note that you cannot have the overlay container open in a separate terminal session while you attempt to use it via snakemake (though it does look like you can run multiple independent jobs simultaneously via snakemake, with the -j flag).

Here's a way to test that all of the above is working:

Copy the following into ~/Snakefile

``` python rule test_run: log: 'test_run.log' run: import numpy print("Success!")

rule test_shell: log: 'test_shell.log' shell: "python -c 'import numpy'; echo success!" ```
From your home directory, run snakemake -j 2 --profile slurm test_run test_shell. If everything was set up correctly, it should run without a problem. If not, check the logs ~/test_run.log and ~/test_shell.log to see if they contain any helpful information. You may also want to add the --verbose flag to the snakemake command, which will cause it to print out the snakemake jobscript to the terminal, making it easier to debug.

VSS 2020 project

2020-06-19T00:00:00-04:00

Introduction

In this post, I'm going to attempt to explain my poster at the Virtual Vision Sciences Society (VSS) conference, 2020, which starts today. The poster itself, along with a video walkthrough, an approximate transcript of that video, and some supplementary images, can be found on an OSF page (view the README.md file for an explanation of the contents) and, if you're a member of VSS, on their website.

Big picture, in this project we study how vision changes across the visual field by investigating what people cannot see. We use computational models of the brain and human behavioral experiments to do this. In this post, I'll first try to give a High-Level Overview of this project, explaining what we do and why it matters. Then, in the Unseen Vision section, I'll spend some more time talking about the approach of studying what cannot see, which has a long history in vision science. I'll then dive into the specific models we use in the Models and the Visual System section, before finishing up and describing the experiment. Unfortunately, because of COVID-19, we have not been able to gather any data yet, but I do have some stimuli that we'll use in the experiment.

High-Level Overview

I am interested in how vision changes across the visual field. You can notice this by observing that, when your eyes are centered on this text, you can read it clearly, but if you move your eyes even a little to the side, you can't make any sense of it. What's going on there? Specifically, we investigate what information people are not sensitive to as you move away from the center of gaze. There are many ways one could approach this, but we're doing so by building models of two of the first stages of the brain that receive visual information: the retina (even though it's in the eye, it's made up of brain tissue and thus technically part of the brain) and the imaginatively-named primary visual cortex (or V1), inspired by our understanding of the cells there¹. Those models have a single parameter, a number we can vary to change their behavior. I'll explain the parameter in more detail in the Models section of this post, but we then generate sets of images that the models (with specific parameter values) think are identical. We then run an experiment where we show those images to humans and find which images they also think are identical, finding the best parameter value for each model.

Once we've done that, we will have two models (with specific parameter values) where we know, if the model thinks two images are identical, humans will not be able to tell them apart ².

Why does that matter? There are several reasons I find this work interesting:

We're modeling specific brain areas, so the parameter has a meaning and therefore tells us something about that brain area. I'll discuss what, exactly, it tells us in the Models section.
Our models were built based on an understanding of cells that comes from showing images to animals (generally, cats or macaques) and recording the activity of those cells using electrodes. Often these images are unlike most of what these or other animals see in their life (black and white gratings or moving bars being the most common). Taking understanding that comes from what happens when animals see something and using it to generate predictions for when humans will not be able to see something is a strong test of this understanding.
We are abstracting away lots of interesting biology about these two brain areas areas and ignoring everything that happens after them³. If we can still generate images that humans find identical, it tells us that these are reasonable assumptions in these conditions (though there are other conditions where that might not be true).
The model can tell us a way to compress the image: the model uses far fewer numbers than there are pixels in the image, and yet those numbers are enough to create an image that humans find identical. This is similar to other image compression methods, like JPEG: when you create a JPEG, it throws away information that it thinks you won't notice, so instead of recording each pixel value for a patch of blue sky, it says "this patch of 100 pixels should all be the same color blue", which takes far less space on your computer. And similar to our models, if you try to throw out too much information, people will notice.
This is a first step towards the creation of what we call a foveated image metric. That's a technical term, so I'll unpack it in parts. An "image metric" tells you how different two images are. The simplest way to do this is to use the mean-squared error, or MSE, where you go through and check how different each pixel value is. But ideally, our measure of how different two images are would map onto how different humans think two images are, and humans and MSE do not agree⁴. You can run experiments to find how different humans think to images are, but that takes a lot of time, so we want to come up with some computer code that can do it for us. With our models, we can say whether humans will think two images look the same, but if the images look different, our models won't tell us how different they look – that's one extension of this project. And "foveated" means that we care about where people are looking in the image; the alternative assumes that people are free to move their eyes everywhere. This point and the last one about compression are related: if you know how similar two images look, you can figure out clever ways to throw away information without changing the perception of the image too much.

Unseen Vision

In my video, I start by discussing the idea that we can study vision by studying what people cannot see, and it's worth talking about this in more detail, because it's a powerful idea, but not an intuitive one.

You're probably aware that color displays, from cathode ray tube screens to liquid crystal displays to projectors, all use three different color primaries (red, green, and blue) to render the colors they show. But how is it that you can use only three different colors to render all of the many different possible colors that humans can view? Because of cones – (most) humans have three types of cones, and so you can get away with only three primaries. But why? And the theory of human trichromatic vision (that humans have three classes of photoreceptors sensitive to color) was first postulated by Thomas Young in 1802, more than 150 years before the physiological evidence for their existence. How?

The existence of three cone classes was theorized as a way to explain the results of color matching experiments, done in the 18th and 19th centuries. In those experiments, participants were shown two lights on either side of a divider. One light, the test light, was a constant color, while the other, the comparison light, was made by combining three different primary lights (for example, red, green, and blue) with different intensities. The participant's task was to adjust the intensities of these three primaries until the two lights looked identical. And people could do this, for any color test light, as long as they had three primaries. And these results were remarkably consistent across people – just about everyone could match colors as long as they had three primaries, and they used the same relative intensities, but they couldn't do it with two⁵.

Brief digression to talk about the nature of color, with the caveat that I do not study color (it's a whole separate area of vision science) and so everything I say should be taken with a grain of salt (if you're interested in this, I recommend reading the chapter on color from Brian Wandell's excellent Foundations of Vision, available for free online). But, technically, color is not a property of an object, it's an interaction between the object (how much light it reflects at each wavelength), the lighting conditions (how much light is present at each wavelength), and the observer (how their visual system interprets the light that arrives at their eye). So, there are no blue dresses, there are just dresses that appear blue to me under specific lighting conditions. That may sound purposefully obtuse, but it's important to keep in mind. If you remember the Dress, a viral image from several years ago, it is a striking example of ambiguity in color perception: some people see it as black and blue, where others see it as white and gold, and people often have very strong opinions on which they see, with few people able to see both⁶. This demonstrates that "color" is not as straightforward a label as we think, so I want to be clear about the differences between perceptual color (what I mean when I say "that dress is blue", which depends on all of the object, lighting, and observer) and the light that arrives at the eye (the amount of energy at each wavelength, known as the power spectrum, which just depends on the object and the lighting).

In the color matching experiment, the two lights were perceived as identical by the participants, but the power spectrum were very different. But the participants didn't notice the difference because their visual system had discarded all information that could separate the two. The two lights are called metamers: they're physically different, but perceptually identical. Like I said earlier, this is because (most) humans have three cone classes. The two lights appear identical because the cone activity for the two of them are the same. They have different amounts of energy at each wavelength, but they excite the cones the same amount, and so you have no way of telling them apart and thus perceive them as identical.

This may seem weird at first – we have a tendency to think of our visual system as conveying accurate information about the world around us. But that's not what it does! It conveys useful information about the world around us, where useful is in evolutionary terms. Think about the electromagnetic spectrum. Our visual system is only sensitive to visible light, not infrared or ultraviolet. But light at those frequencies are still present in the world and arrive at our eyes – we just can't make use of it because our visual system throws it away. Our cones only respond to lights between 400 and 700 nanometers, and so we cannot tell the difference between a blue dress and a blue dress with a UV lamp behind it. Other animals' cones, however, like bees and some birds, are sensitive to ultraviolet light, and so their visual system can make use of it.

So alright, human visual systems aren't perfect and can be tricked in this fairly arbitrary way. What of it? Well, now I can describe a color using only three numbers, the intensities of each of those three primary lights, rather than requiring me to specify the full power spectrum, which would require a number for each possible wavelength in the visible spectrum. That's a lot less information! I've gone from 300 numbers to just 3. This is why I only need three color primaries in a display – I can't reproduce any possible power spectrum, but I can match the perceived color pretty well. It would be much harder to fit all the necessary lights into a screen if we needed 300 of them. I've made use of my understanding of the human visual system to compress the information about the color. This is one important application of this type of work: if we know what information the human visual system throws out, we can throw away that same information in any thing we build that interacts with the human visual system (you might want to hold onto it for other reasons, but for most situations where you just want an image to look good, it's fine).

And note that this whole thing is based around investigating what humans don't see. We've found changes we can make to the physical stimulus (in this case, the power spectra) without humans being able to tell that anything is different. This isn't about what colors look like: why do some colors complement each other? What makes a color stand out? How hard are these two colors to tell apart? There's a whole host of interesting questions there as well. In the color case, the experiments to find what people cannot see preceded the theory about the visual system that explains why. But these types of experiments can also serve as a strong test of our theories and understandings of the visual system: we should be able to use our understand of how the system works to generate images humans cannot distinguish. And we should be able to do this not only by throwing information out of an image and predicting you can't tell, but also by adding new information that you won't be able to see. That's the goal of this project, to build models of early stages of the visual system based on our understanding from other experiments, generate images that the models think are identical, and run an experiment to see which images humans also think are identical.

Models and the Visual System

With the previous section, I hope I've shown how studying what people don't see can be a useful way to increase our understanding of the visual system. In this project, we use that approach to study how vision changes across the visual field. This is a pretty drastic effect: when your eyes are fixated on this text, you can read it without a problem, but if you move your eyes to the edge of the computer screen, then the text becomes illegible. Why would this be the case?

First, let's talk a little about the layout of the visual system. I'm going to talk about the flow of information in the brain: information enters via our sensory systems and gets processed and transformed by a series of connected brain areas. These transformations, or computations, are what my research focuses on. How can we recognize faces so easily from an image, which is just a bunch of points of light? Because the human brain transforms those points into some more meaningful representation that allows us to easily determine how "face-like" something is. But the fact that it has taken years to get computers to be able to recognize faces with any accuracy (and they're still not that good, and have all sorts of biases) should emphasize how hard this is. To return to the anatomy: when light enters the eye, it travels through the lens to the back of the eye, called the retina. From there, the information gets sent to a section of the thalamus, deep in the brain, called the lateral geniculate nucleus, or LGN (generally speaking, in visual neuroscience we tend to consider the LGN a relay station that doesn't transform the information at all, and so ignore it). From there, it goes to the primary visual cortex or V1, before going through a succession of other similarly-named areas: V2, V3, V4, etc. In this project, we build models of V1 and the retina.

The visual field is what you're seeing at any given time. When we discuss the visual field, we're less interested in the physical objects out in the world than we are in the patterns of light that these objects reflect or emit that land on your eye. The center of your visual field, where your eyes are fixating at any one point in time, is the fovea. The term comes from the Latin word for "pit" and describes an interesting anatomical feature of the retina: the cells in front of the cones are shoved to the side so they don't get in the way of the light. This is the region of highest acuity. As you move away from the fovea, you eventually enter the para-fovea, and then end up in the periphery, your acuity decreasing gradually all the way⁷⁸. These terms are used to refer to the anatomy of the retina and later brain areas as well as to the perception of that part of the visual field. So if we're discussing what you see where your eyes are centered, that's "foveal vision" or just the "fovea". Finally, when talking about locations in the visual field, the distance from the fovea is the eccentricity and it's measured in degrees of visual angle, such that you'd say "the image was presented at 10 degrees eccentricity" when describing the stimuli in an experiment.

The visual system is a portion of the brain (in primates, quite a large portion!) and so, like the rest of the brain, is made up of specialized cells called neurons. As a computational neuroscientist, I don't think so much about all the interesting biology of neurons, and really only think about them in terms of their activity. People spend their entire lives studying neuronal activity, but for my purposes all we need to know is that it's the way neurons communicate with each other, so if something isn't reflected in neuronal activity, the visual system doesn't know about it. To return to our color example, ultraviolet light has no effect on cone activity, and so the visual system knows nothing about it.

If we want to understand the visual system, we should understand what makes neurons active and how that activity changes. One of the foundational results in visual neuroscience was the discovery of receptive fields in the 1950s and 1960s by David Hubel and Torsten Wiesel. They found that neurons in cat V1 got active when they moved a bar into a certain portion of the visual field, and were not active when the bar was in any other portion of space⁹. This is the neuron's receptive field, the portion of the visual field that the neuron cares about¹⁰. These receptive fields grow larger as you move away from the fovea and also as you go deeper into the visual system, so that foveal retina receptive fields are tiny (the size of a cone), whereas peripheral V4 receptive fields are giant (a quarter of your visual field). In the beginning of this section, I pointed out that your vision gets worse as you move away from the fovea, and the way it does this seems a lot like it's "getting bigger": things in your periphery appear somewhat blurry and hard to distinguish from each other. That seems like it might be related to receptive fields growing larger as you move away from the fovea, so that's what we're going to focus on. There may be other important differences between the fovea and periphery, but for this project, the only difference is that receptive fields have grown larger.

Visual neurons are sensitive to more than just a region of space. Hubel and Wiesel found neurons that only responded if the bar was in a certain portion of space and had a certain orientation (such as vertical or horizontal). This led to the idea that neurons are sensitive to "features" as well as locations¹¹. In V1, the main features are orientation and size¹², and across all of V1, they, along with location, are all represented. This information about orientation and size is called spectral energy. We know each orientation is represented everywhere in the visual field, but size isn't (there aren't neurons that respond to small things in the periphery and neurons responding to big things might not be found in the fovea¹³). However, for our models, we're going to say all orientations and all sizes are represented everywhere in the visual field¹⁴. Different visual areas care about different things. Retina does not care about orientation or size: for our purposes, it only cares about brightness¹⁵¹⁶. These things that the brain areas care about are called summary statistics.

Those are the core ideas for our models: brain areas compute summary statistics and the main thing that changes with eccentricity is the size of these receptive fields. So let's put them together: what do we do with those summary statistics in those receptive fields? Let's take the simple approach and just average them. So, a neuron's activity is based on how much of its favorite stuff is in its favorite area of the visual field. A given neuron in V1, for example, might care about how much "vertical-ness" of a certain size there is in a given patch of the visual field, and measuring that in an image is a pretty good way to predict how active that neuron will be when the animal is shown the image. A given neuron in the retina, on the other hand, might care how bright a certain patch of the visual field is. Note that the neurons are ignoring the details that gave rise to those statistics (the retina neuron just cares about the average brightness in its favorite area, not whether that brightness came from a really bright object and a really dark object, or two medium-bright objects) – which means we're throwing away information! This is where we return to the concept I discussed in Unseen Vision: if we're throwing away information, that means we can find metamers. Also note that the amount of information the model throws away increases both as the receptive field grows and as the number of statistics shrink, and these happen independently¹⁷. I'll return to this idea when I describe the experiment in more detail in the next section.

When we build these models, we have two choices: the statistics they calculate and the size of the windows (I'm going to refer to them as "pooling windows" from now on, and use "receptive fields" only to refer to actual neurons) they average them in. We pick the statistics based on our understanding of the visual system. This is important – in no part of this study do we test different statistics to see which is best. We build in the assumption (based on decades of research) that we can summarize an important part of neuronal activity in the retina and V1 by using brightness and spectral energy. We're also going to say that window size grows with eccentricity¹⁸, and the only thing we need to find in our experiment is how quickly it grows. This scaling, as we call it, is the model's only free parameter, i.e., it's the only thing we fit to the data. Finding it is the goal of the experiment that we have not yet had a chance to run, because of COVID.

To summarize, in this project, we built models of the retina and V1, which average summary statistics in windows that grow larger with eccentricity. To see how well they model human perception, we generated model metamers, images that are physically different but that the model thinks are identical. In the next section, I'll discuss the experiment we're planning on running in order to test those models.

Experiment

Still working on this!

There's a part of the brain betwen the retina and V1, a region of the thalamus, deep in the brain, called the lateral geniculate nucleus. However, in visual neuroscience we often consider it to be a relay station that just transmits signals between the retina and V1, so we ignore it in this project ↩
At least, under the conditions of our experiment. ↩
This is known as a feed-forward approach to the brain, assuming information flows through the brain in one direction and ignoring all the feedback that happens. ↩
See this paper from Zhou Wang and Alan Bovik for more details. ↩
There were some people, it turns out, who could do it with two primaries – people with dichromacy, a form of color blindness). ↩
The current understanding is that it comes down to differences in what you (implicitly) assumed the lighting condition was when the photo was taken. If you thought the light was yellow-tinted, you'll see the dress as black and blue; if you thought it was blue-tinted, you'd see the dress as white and gold. Wikipedia has a decent explanation on this. ↩
To my knowledge, there are no agreed upon boundaries for any of these terms, except for the fovea; there is an anatomical boundary where the actual pit of the fovea ends. ↩
Interestingly, not all animals have foveas, and some birds have two! It seems to be present in animals that need precise information about the location of objects, like primates, birds, and cats. But prey animals, such as mice and rabbits, do not have foveas, and often have quite poor visual acuity. Again, this comes down to the fact that the visual system is about capturing useful information to the organism, and fine spatial information is not important to mice. ↩
You can find footage of this experiment on YouTube (the clicking noise is the neurons firing action potentials, and the more often that happens, the more active they are). ↩
The concept comes from Sherrington, who used it to describe the area of skin from which a scratch reflex could be elicited in a dog. ↩
I put "features" in scare quotes because I have lots of feelings about the features neurons are responsive to – they're often not nearly as human-interpretable as we'd like, and I think the idea that V1 neurons are "edge-detectors", a common interpretation of Hubel and Wiesel's studies (including by the original authors), has led to a lot of confusion in visual neuroscience. But that's outside the scope of this post. Though I do recommend reading Adelson and Bergen's great paper on The Plenoptic Function if you're interested in this. ↩
Technically, spatial frequency, but for the purposes of this post we can call it size. ↩
This is actually the topic of the first major project I worked on in grad school, as presented at VSS 2018. I still haven't finished it yet, however, because science is slow. ↩
Changing this is one of the extensions of this project that we're working towards: removing unnecessary information about size / spatial frequency based on location in the visual field. ↩
Folks who actually study the retina will not like this characterization, which ignores lots of interesting stuff that happens in the retina, but it's enough for our purposes. ↩
Later areas in the visual system get much more complicated quickly, and there's not nearly as much agreement on them as there is about V1. ↩
By which I mean, for a given number of statistics, you'll throw away more information as the receptive fields grow, and, for a given size of receptive field, you'll throw away more information as the number of statistics shrinks. ↩
Specifically, based on a literature review shown in Figure 1a in this paper, we're going to say that their width grows linearly with eccentricity ↩

Emacs python setup

2019-01-01T00:00:00-05:00

I got a new laptop recently and so set up Emacs on it for the first time. Since my emacs init.el file has gotten progressively messier over the past several years, I took the opportunity to think about the variety of python packages I use and set things up cleanly. The following is what I settled on; the steps involved weren't as clear as I had hoped, so I'm writing this all down in case I need to do it again.

Some notes on how I use Emacs and python before I start:

I use conda to manage my virtual environments. Most projects, but not all, have a corresponding environment.yml file in their repository. I installed miniconda at ~/miniconda3 (conda version 4.5.12).
I do the early parts of my development, the playing around with ideas before I start writing scripts, in a Jupyter Notebook. I also do the end of my analysis in them, creating plots and putting the parts together in them. I keep most of my functions in scripts, runnable from the command line so that I can submit them to NYU's computing cluster. Therefore, I don't need the full IDE experience; I don't need to run a shell session from within Emacs, and I prefer using Jupyter to ein, org-babel, or anything similar.
I primarily use el-get to manage my Emacs packages, falling back on use-package when an appropriate el-get recipe isn't available. My configurations for both of these are very basic (and I installed use-package using el-get).

Because I don't need the full IDE experience and want to use conda to manage my virtual environments, I cannot use Elpy, which combines a variety of python-related packages together with a minimum of setup difficulty (here's a guide to getting started with it). So I decided on using jedi.el (with the company backend) for auto-completion, the included python-mode (in Emacs 26.1) for basic syntax highlighting, conda.el for managing virtual environments, and flycheck (with pylint and flake8) for syntax- and style-checking. Wanting to use conda and the company backend meant things were slightly more complicated to set up.

First, conda.el. This was very straightforward to setup just following the instructions on the Github page. The following is the relevant block in my init.el:

(use-package conda
  :ensure t)
(require 'conda)
;; if you want auto-activation (see below for details), include:
(conda-env-autoactivate-mode t)
(custom-set-variables
 '(conda-anaconda-home "~/miniconda3"))

Next, company-mode and jedi.el, both of which I can install using el-get. (Note that, if you want to use jedi with the company backend, do not install the regular jedi.el: you should only install company-jedi, see this issue).

(setq my:el-get-packages
      '(company-mode))
(el-get-bundle elpa:jedi-core)
(el-get-bundle company-jedi :depends (company-mode))
(eval-after-load "company-jedi"
    '(setq jedi:server-command (list "~/miniconda3/envs/emacs-jedi/bin/python" jedi:server-script)))
(require 'company-jedi)
(el-get 'sync my:el-get-packages)

However, I can't use the automatic jedi:install-server command, and so need to do some manual set up. I mostly followed the instructions from Update 2 of this stackoverflow answer, with some changes:

Create a conda environment (for current example the environment is named emacs-jedi) by doing: conda create -n emacs-jedi python
Install the following python packages: jedi, sexpdata, epc. Only jedi is on conda, so I did the following: pip install sexpdata epc; conda install jedi.
Install the jediepcserver. Navigate to the jedi-core install directory (probably ~/.emacs.d/el-get/jedi-core; it should contain jediepcserver.py), then run: python setup.py install (note the directory has to exist, so the (el-get-bundle elpa:jedi-core) needs to be run before this).

After doing the above, company-jedi should now be setup. The following are the relevant config blocks in my init.el

(add-hook 'conda-postactivate-hook 'jedi:stop-server)
(add-hook 'conda-postdeactivate-hook 'jedi:stop-server)

(defun my/python-mode-hook ()
  (add-to-list 'company-backends 'company-jedi))

(add-hook 'python-mode-hook 'my/python-mode-hook)

(add-hook 'python-mode-hook 'jedi:setup)
(setq jedi:complete-on-dot t)

Finally, install and set-up flycheck. This is relatively straightforward: flycheck can be installed using el-get:

(setq my:el-get-packages
      '(flycheck))
(el-get 'sync my:el-get-packages)

Now, install the pylint and flake8 executables, in the base conda environment:

conda activate base
pip install flake8 pylint

And tell flycheck where to find the executables (and add a couple extra lines of configuration):

(add-hook 'after-init-hook #'global-flycheck-mode)
(setq-default flycheck-emacs-lisp-load-path 'inherit)
(setq flycheck-flake8-maximum-line-length 99)
(setq flycheck-python-pylint-executable "~/miniconda3/bin/pylint")
(setq flycheck-python-flake8-executable "~/miniconda3/bin/flake8")

Once that's done, all you need to do is check things are set up. Open up any python file in Emacs and type C-u C-c ! v to see the status of flycheck and, if pylint and flake8 are deactivated, type C-u C-c ! x to activate them.

If everything works, flycheck will start underlining things in yellow and red to tell you that things are against the style guide or will raise errors. If you start to type something, jedi will suggest possible completions, and all of these will be based on the packages installed in the appropriate virtual environment.

All told, the following shows the relevant parts of my init file:

(setq my:el-get-packages
      '(company-mode
        flycheck))
(el-get-bundle elpa:jedi-core)
(el-get-bundle company-jedi :depends (company-mode))
(eval-after-load "company-jedi"
    '(setq jedi:server-command (list "~/miniconda3/envs/emacs-jedi/bin/python" jedi:server-script)))
(require 'company-jedi)
(el-get 'sync my:el-get-packages)

(use-package conda
  :ensure t)
(require 'conda)
;; if you want auto-activation (see below for details), include:
(conda-env-autoactivate-mode t)
(custom-set-variables
 '(conda-anaconda-home "~/miniconda3"))
(add-hook 'conda-postactivate-hook 'jedi:stop-server)
(add-hook 'conda-postdeactivate-hook 'jedi:stop-server)

(defun my/python-mode-hook ()
  (add-to-list 'company-backends 'company-jedi))

(add-hook 'python-mode-hook 'my/python-mode-hook)
(add-hook 'python-mode-hook 'jedi:setup)
(setq jedi:complete-on-dot t)

(add-hook 'after-init-hook #'global-flycheck-mode)
(setq-default flycheck-emacs-lisp-load-path 'inherit)
(setq flycheck-flake8-maximum-line-length 99)
(setq flycheck-python-pylint-executable "~/miniconda3/bin/pylint")
(setq flycheck-python-flake8-executable "~/miniconda3/bin/flake8")

Accidental illusion

2017-12-14T00:00:00-05:00

NOTE: You can see a webapp I put together to investigate this in more detail here.

While creating two-dimensional sinusoidal gratings and investigating their local properties, I accidentally created an illusion today, which I find pretty strong (though the strength of it does depend on its size and your distance to it).

I created the grating shown below:

and was dropping down some circular windows on it to see if I could recreate the same image by making a bunch of small sinusoids and combining them. What I ended up seeing was:

To me, it looks like there's an illusory grating that's perpendicular to the actual one, but if you focus on the location of those supposed stripes, putting them in the center of your gaze, they disappear. This is pretty strong to me, even overwhelming the original grating unless I focus directly on it. It changes with size / distance though, so that if I make it really small or stand far away, it looks more cross-hatched, a combination of the two patterns, than either one alone.

I'm not entirely sure why this is happening (and, discussing it with the lab, neither is anyone else). It seems similar to the following classic illusion, where the Xs in either side of the image look like their yellow / blue, respectively, but are actually the same color.

A similar illusion is shown below, where the two squares are the same grayscale level, but appear to be different.

In all of these, the illusion relies on the fact that our perception of brightness and color of an object is influenced by the contrast with the stuff around that object and not just the object itself. This relates to a recurring theme in vision and perception more generally: we are not just passively perceiving a perfect copy of the world, our sensory systems perform computation on our input, which highlights some pieces of information and throws away other pieces. This happens so fast and automatically that we often don't realize it's happening until we see illusions like the above, which make it obvious.

William Broderick's Blog

snakemake, singularity, and HPC

build_docker:

singularity_env.sh

config.json

run_singularity.py

slurm snakemake profile

Archiving

Conclusion

Scripting figure creation with inkscape and python

conda, snakemake, and HPC

if you're not on NYU's Greene cluster, this path will almost certainly be

different

again, if you're not on Greene, the path to the linux singularity image will

almost certainly be different

!/bin/bash

this needs to be copied into the overlay, as /ext3/env.sh

set up other libraries (module doesn't work in the container)

exit the container

create the snakemake slurm profile