forked from ZTF-Summer-School/ztf_summer_school_2022
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
15 changed files
with
3,091 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,103 @@ | ||
| # ztf_summer_school_2022 | ||
| Repository for the ZTF Summer School 2022 | ||
| # ZTF Summer School 2022 | ||
| ### Hosted hybrid by the University of Minnesota | ||
| #### July 25-29: 9 am - 12:30 pm (synchronous), group work in the afternoons (asynchronous) | ||
|
|
||
| Course materials for ZTF Summer School 2022 | ||
|
|
||
| ### Purpose | ||
|
|
||
| This course will introduce key concepts and techniques used to work with large datasets, in the context of the field of astrophysics. In the first 3 days of the course, the focus will be on the modern approaches to creating and manipulating large data sets, with the focus on time series analyses and Bayesian methods applied to astrophysics survey data. The remaining part of the course will focus on a range of machine learning techniques for processing data: classification algorithms (supervised and unsupervised learning), clustering algorithms, regression problems, recommender systems, graphic models and others. The algorithms will first be introduced in lectures, and the emphasis will then be placed on homework worked as teams in which the students will apply the algorithms (and already available packages) to astrophysical data sets to answer specific astrophysics questions. The course will assume familiarity with basic concepts in astrophysics, but it will include brief reviews as needed to demonstrate the use of modern data analysis techniques. | ||
|
|
||
| ### Sample Syllabus | ||
|
|
||
| Day 1: | ||
| * Talks: ZTF Introduction, LIGO / GW Follow-up | ||
| * Tutorials: Alert Stream Access, Bayesian Inference | ||
|
|
||
| Day 2: | ||
| * Talks: Searches for fast transients with ZTF, Gravitational-wave data products / observing scenarios | ||
| * Tutorials: Fast transient case study, interacting with skymaps | ||
|
|
||
| Day 3: | ||
| * Talks: Fermi-GBM Follow-up, Neutrino Follow-up | ||
| * Tutorials: Gaussian Process kilonova grids, Neutrino searches | ||
|
|
||
| Day 4: | ||
| * Talks: Introduction to Machine Learning for MMA, Introduction to Marshals | ||
| * Tutorials: Introduction to Machine Learning, Kilonova / GRB sample case study | ||
|
|
||
| Day 5: | ||
| * Talks: Introduction to Brokers, Automated spectral classification | ||
| * Tutorials: Introduction to Deep Learning, Real-Bogus case study | ||
|
|
||
| ### How to use these materials | ||
|
|
||
| * Fork this repo | ||
|
|
||
| * The topics of each day's lectures are described in the schools' program | ||
|
|
||
| * Lectures are in directories named XY/ where XY/ is the number of the day | ||
|
|
||
| * Homework assignments are similarly in the homework/ directory | ||
|
|
||
| * Solutions to homework assignments will be posted the day after they are due. | ||
|
|
||
| * Various help cheat sheets are included in help/. | ||
|
|
||
| * You should also review in-class notebooks and homework solutions to make sure you understand what is happening | ||
|
|
||
| * The lecture notebooks have in-class exercises for you to work on | ||
|
|
||
| ### Related Material | ||
|
|
||
| * UIUC Fundamentals of Data science: https://github.com/gnarayan/ast596_2020_Spring | ||
| * Caltech Astroinformatics: https://sites.astro.caltech.edu/ay119/ | ||
| * GROWTH summer school: http://growth.caltech.edu/growth-school-2019.html | ||
| * AURA winter school: http://www.aura-o.aura-astronomy.org/winter_school/ - go to Past Years. | ||
| * YouTube Neural Networks: https://www.youtube.com/watch?v=aircAruvnKk | ||
| * UMN Big Data Astronomy: https://github.com/mcoughlin/ast8581_2022_Spring | ||
|
|
||
| ### Github | ||
| If you are new to github, we recommend watching: | ||
| * https://www.youtube.com/watch?v=USjZcfj8yxE&ab_channel=ColtSteele | ||
| * https://www.youtube.com/watch?v=nhNq2kIvi9s&ab_channel=ColtSteele | ||
|
|
||
| ### Environments | ||
| If running locally, we suggest using anaconda: https://anaconda.org/ | ||
|
|
||
| If you are on windows, use the windows subsystem for linux: | ||
| https://docs.microsoft.com/en-us/windows/wsl/install-win10 | ||
| plus the ubuntu anaconda. | ||
|
|
||
| After installing anaconda, you should (in a terminal): | ||
|
|
||
| * clone the repository: git clone [email protected]:mcoughlin/ztf_summer_school_2022.git | ||
| * change directories to the repository: cd ztf_summer_school_2022 | ||
| * use conda to create a repository | ||
| * on mac osx: conda env create -f environment-osx.yml | ||
| * on linux / windows wsl: conda env create -f environment-ubuntu.yml | ||
|
|
||
| Activate the environment with: | ||
| * source activate ztfsummer | ||
| and can test by opening the first lecture: | ||
| * cd lectures/01 | ||
| * jupyter notebook lecture01.ipynb | ||
|
|
||
| Either will, you will need to pip install: | ||
| * pip install git+git://github.com/colcarroll/minimc.git | ||
|
|
||
| Otherwise, google colaboratory will work: | ||
| https://colab.research.google.com/ | ||
|
|
||
| Will need to create an IRSA account for data access: | ||
| https://irsa.ipac.caltech.edu/frontpage/ | ||
|
|
||
| # ZTF useful links + QA | ||
| | Question | Answer | | ||
| | ------------- | ------------- | | ||
| | ztfquery documentation | https://github.com/MickaelRigault/ztfquery | | ||
| | ZTF table content | https://irsa.ipac.caltech.edu/onlinehelp/ztf/overview.html | | ||
| | Alerts content | https://zwickytransientfacility.github.io/ztf-avro-alert/schema.html | | ||
| |What is the "Redshift Completeness Factor" program? | It's a program dedicated to determine the number of SN host galaxies with known spectroscopic redshift prior to the SN discovery divided by the total number of SN hosts. See https://arxiv.org/abs/1910.12973 | | ||
| |What is the "Census of the Local Unverse" program? | The Census of the Local Universe (CLU) aims to provide a galaxy catalog out to 200 Mpc that is as complete as possible. See https://arxiv.org/abs/1710.05016 | | ||
| |How is the time allocation for each filter decided? | ZTF uses a Integer Linear Programming approach, that optimizes an observing plan for an entire night by assigning targets to temporal blocks, enables strict control of the number of exposures obtained per field and minimizes filter changes. See https://arxiv.org/abs/1905.02209 | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,244 @@ | ||
| name: ztfsummer | ||
| channels: | ||
| - pytorch | ||
| - astropy | ||
| - conda-forge | ||
| - defaults | ||
| dependencies: | ||
| - _libgcc_mutex=0.1 | ||
| - _openmp_mutex=4.5 | ||
| - _tflow_select=2.3.0 | ||
| - absl-py=0.13.0 | ||
| - aiohttp=3.7.4.post0 | ||
| - argon2-cffi=20.1.0 | ||
| - arviz=0.11.2 | ||
| - astor=0.8.1 | ||
| - astroml=0.4.1 | ||
| - astropy=4.3.post1 | ||
| - astunparse=1.6.3 | ||
| - async-timeout=3.0.1 | ||
| - async_generator=1.10 | ||
| - attrs=21.2.0 | ||
| - autograd=1.3 | ||
| - backcall=0.2.0 | ||
| - blas=1.0 | ||
| - bleach=4.0.0 | ||
| - blinker=1.4 | ||
| - bottleneck=1.3.2 | ||
| - brotli=1.0.9 | ||
| - brotlipy=0.7.0 | ||
| - c-ares=1.17.1 | ||
| - ca-certificates=2021.7.5 | ||
| - cached-property=1.5.2 | ||
| - cachetools=4.2.2 | ||
| - cairo=1.14.12 | ||
| - certifi=2021.5.30 | ||
| - cffi=1.14.0 | ||
| - cftime=1.5.0 | ||
| - chardet=4.0.0 | ||
| - charset-normalizer=2.0.0 | ||
| - click=8.0.1 | ||
| - colorama=0.4.4 | ||
| - corner=2.2.1 | ||
| - cryptography=3.4.7 | ||
| - cudatoolkit=10.2.89 | ||
| - curl=7.78.0 | ||
| - cycler=0.10.0 | ||
| - dataclasses=0.8 | ||
| - dbus=1.13.18 | ||
| - decorator=5.0.9 | ||
| - defusedxml=0.7.1 | ||
| - emcee=3.1.0 | ||
| - entrypoints=0.3 | ||
| - expat=2.4.1 | ||
| - fontconfig=2.13.1 | ||
| - fonttools=4.25.0 | ||
| - freetype=2.10.4 | ||
| - fribidi=1.0.10 | ||
| - future=0.18.2 | ||
| - gast=0.4.0 | ||
| - gatspy=0.3 | ||
| - glib=2.63.1 | ||
| - google-auth=1.34.0 | ||
| - google-auth-oauthlib=0.4.1 | ||
| - google-pasta=0.2.0 | ||
| - graphite2=1.3.14 | ||
| - graphviz=2.40.1 | ||
| - grpcio=1.38.1 | ||
| - gst-plugins-base=1.14.0 | ||
| - gstreamer=1.14.0 | ||
| - h5py=2.10.0 | ||
| - harfbuzz=1.8.8 | ||
| - hdf4=4.2.13 | ||
| - hdf5=1.10.6 | ||
| - icu=58.2 | ||
| - idna=3.1 | ||
| - importlib-metadata=3.10.0 | ||
| - importlib_metadata=3.10.0 | ||
| - intel-openmp=2021.3.0 | ||
| - ipykernel=5.3.4 | ||
| - ipython=7.26.0 | ||
| - ipython_genutils=0.2.0 | ||
| - ipywidgets=7.6.3 | ||
| - jedi=0.18.0 | ||
| - jinja2=3.0.1 | ||
| - joblib=1.0.1 | ||
| - jpeg=9b | ||
| - jsonschema=3.2.0 | ||
| - jupyter=1.0.0 | ||
| - jupyter_client=6.1.12 | ||
| - jupyter_console=6.4.0 | ||
| - jupyter_core=4.7.1 | ||
| - jupyterlab_pygments=0.1.2 | ||
| - jupyterlab_widgets=1.0.0 | ||
| - keras-preprocessing=1.1.2 | ||
| - kiwisolver=1.3.1 | ||
| - krb5=1.19.2 | ||
| - lcms2=2.12 | ||
| - ld_impl_linux-64=2.35.1 | ||
| - libcurl=7.78.0 | ||
| - libedit=3.1.20210216 | ||
| - libev=4.33 | ||
| - libffi=3.2.1 | ||
| - libgcc-ng=9.3.0 | ||
| - libgfortran-ng=7.5.0 | ||
| - libgfortran4=7.5.0 | ||
| - libgomp=9.3.0 | ||
| - libnetcdf=4.6.1 | ||
| - libnghttp2=1.41.0 | ||
| - libpng=1.6.37 | ||
| - libprotobuf=3.17.2 | ||
| - libsodium=1.0.18 | ||
| - libssh2=1.9.0 | ||
| - libstdcxx-ng=9.3.0 | ||
| - libtiff=4.2.0 | ||
| - libuuid=1.0.3 | ||
| - libuv=1.40.0 | ||
| - libwebp-base=1.2.0 | ||
| - libxcb=1.14 | ||
| - libxml2=2.9.12 | ||
| - lz4-c=1.9.3 | ||
| - markdown=3.3.4 | ||
| - markupsafe=2.0.1 | ||
| - matplotlib=3.4.2 | ||
| - matplotlib-base=3.4.2 | ||
| - matplotlib-inline=0.1.2 | ||
| - mistune=0.8.4 | ||
| - mkl=2021.3.0 | ||
| - mkl-service=2.4.0 | ||
| - mkl_fft=1.3.0 | ||
| - mkl_random=1.2.2 | ||
| - mongodb=4.0.3 | ||
| - mpi=1.0 | ||
| - mpi4py=3.0.3 | ||
| - multidict=5.1.0 | ||
| - munkres=1.1.4 | ||
| - nbclient=0.5.3 | ||
| - nbconvert=6.1.0 | ||
| - nbformat=5.1.3 | ||
| - ncurses=6.2 | ||
| - nest-asyncio=1.5.1 | ||
| - netcdf4=1.5.7 | ||
| - ninja=1.10.2 | ||
| - notebook=6.4.2 | ||
| - numexpr=2.7.3 | ||
| - numpy=1.20.3 | ||
| - numpy-base=1.20.3 | ||
| - oauthlib=3.1.1 | ||
| - olefile=0.46 | ||
| - openjpeg=2.3.0 | ||
| - openmpi=4.0.5 | ||
| - openssl=1.1.1k | ||
| - opt_einsum=3.3.0 | ||
| - packaging=21.0 | ||
| - pandas=1.3.1 | ||
| - pandocfilters=1.4.3 | ||
| - pango=1.42.4 | ||
| - parso=0.8.2 | ||
| - patsy=0.5.1 | ||
| - pcre=8.45 | ||
| - pexpect=4.8.0 | ||
| - pickleshare=0.7.5 | ||
| - pillow=8.3.1 | ||
| - pip=21.0.1 | ||
| - pixman=0.40.0 | ||
| - prometheus_client=0.11.0 | ||
| - prompt-toolkit=3.0.17 | ||
| - prompt_toolkit=3.0.17 | ||
| - protobuf=3.17.2 | ||
| - ptyprocess=0.7.0 | ||
| - pyasn1=0.4.8 | ||
| - pyasn1-modules=0.2.7 | ||
| - pycparser=2.20 | ||
| - pyerfa=2.0.0 | ||
| - pygments=2.9.0 | ||
| - pyjwt=2.1.0 | ||
| - pymongo=3.12.0 | ||
| - pyopenssl=20.0.1 | ||
| - pyparsing=2.4.7 | ||
| - pyqt=5.9.2 | ||
| - pyrsistent=0.17.3 | ||
| - pysocks=1.7.1 | ||
| - python=3.8.2 | ||
| - python-dateutil=2.8.2 | ||
| - python-flatbuffers=2.0 | ||
| - python-graphviz=0.16 | ||
| - python_abi=3.8 | ||
| - pytorch=1.9.0 | ||
| - pytz=2021.1 | ||
| - pyu2f=0.1.5 | ||
| - pyyaml=5.4.1 | ||
| - pyzmq=22.2.1 | ||
| - qt=5.9.7 | ||
| - qtconsole=5.1.0 | ||
| - qtpy=1.9.0 | ||
| - readline=8.1 | ||
| - requests=2.26.0 | ||
| - requests-oauthlib=1.3.0 | ||
| - rsa=4.7.2 | ||
| - schwimmbad=0.3.2 | ||
| - scikit-learn=0.24.2 | ||
| - scipy=1.6.2 | ||
| - seaborn=0.11.1 | ||
| - send2trash=1.5.0 | ||
| - setuptools=52.0.0 | ||
| - sip=4.19.13 | ||
| - six=1.16.0 | ||
| - sqlite=3.36.0 | ||
| - statsmodels=0.12.2 | ||
| - tensorboard=2.6.0 | ||
| - tensorboard-data-server=0.6.0 | ||
| - tensorboard-plugin-wit=1.8.0 | ||
| - tensorflow=2.4.1 | ||
| - tensorflow-base=2.4.1 | ||
| - tensorflow-estimator=2.5.0 | ||
| - termcolor=1.1.0 | ||
| - terminado=0.9.4 | ||
| - testpath=0.5.0 | ||
| - threadpoolctl=2.2.0 | ||
| - tk=8.6.10 | ||
| - tornado=6.1 | ||
| - tqdm=4.62.1 | ||
| - traitlets=5.0.5 | ||
| - typing-extensions=3.10.0.0 | ||
| - typing_extensions=3.10.0.0 | ||
| - urllib3=1.26.6 | ||
| - wcwidth=0.2.5 | ||
| - webencodings=0.5.1 | ||
| - werkzeug=2.0.1 | ||
| - wheel=0.37.0 | ||
| - widgetsnbextension=3.5.1 | ||
| - wrapt=1.12.1 | ||
| - xarray=0.19.0 | ||
| - xz=5.2.5 | ||
| - yaml=0.2.5 | ||
| - yarl=1.6.3 | ||
| - zeromq=4.3.4 | ||
| - zipp=3.5.0 | ||
| - zlib=1.2.11 | ||
| - zstd=1.4.9 | ||
| - pip: | ||
| - fastavro==1.4.4 | ||
| - minimc==0.0.1 | ||
| - ztfquery==1.15.9 | ||
| prefix: /home/mcoughlin/anaconda3/envs/ztfsummer |
Oops, something went wrong.