Installation

Docker

This pages walks you through running YSE_PZ locally. You might want to do this if you want to develop on YSE_PZ. The strongly recommended (and only supported) way to run YSE_PZ locally is to use docker. You could attempt to install YSE_PZ natively (instructions at the bottom of this page for archival reasons) but this will likely be an order of magnitude harder than using docker.

Install the Docker desktop app

The first step is to install the docker desktop application, which can be found here for your system. The recommended docker resources required to run YSE_PZ are:

  • System memory: >= 8 GB

  • Docker memory: >= 4 GB

  • Docker CPU >= 1

  • Docker Swap >= 1 GB

  • Docker Disk Image Size >= 20 GB

These requirements can be set in the docker desktop app (Preferences > Resources).

Settings file

In order for YSE_PZ to work, a settings.ini needs to be created in the YSE_PZ/YSE_PZ/ directory. This file contains all the YSE_PZ configuration settings and can also contain secrets needed to access external services. To create a minimum working settings file, you can just copy and rename the public_settings.ini file. From the base YSE_PZ directory run,

cp YSE_PZ/public_settings.ini YSE_PZ/settings.ini

Environment file

For docker compose to run YSE_PZ environment variables like systems ports and volumes need to be set. This is done by creating a .env file in the YSE_PZ/docker directory.

The .env should have the following variables set:

  • VOL

The local path to the root of this repository – will be mapped to /app in the docker web image

  • VOL_DB

The local path to the mysql files, e.g.: “local proj path/docker_mysql/8.0”

  • DB_PWD

The root database password

  • STATIC_VOL

The path YSE_PZ app’s static directory

  • LOCAL_DB_HOST

Configurable - this should be set to whatever port you want docker forwarding it’s database container port 3306 on.

  • LOCAL_HTTP_PORT

Configurable - this should be set to whatever port you want docker forwarding it’s nginx container port 80 on.

  • DJANGO_SUPERUSER_USERNAME

This is the django and yse_pz superuser username that will be created on startup. You use this to log into the YSE_PZ website and the django admin dashboard.

  • DJANGO_SUPERUSER_PASSWORD

This is the django and yse_pz superuser password that will be created on startup. You use this to log into the YSE_PZ website and the django admin dashboard.

  • DJANGO_SUPERUSER_EMAIL

This is the django and yse_pz superuser email that will be created on startup.

An example of a minimum working .env file would be

VOL= ../
VOL_DB=../database/
DB_PWD=password
STATIC_VOL=../YSE_PZ/static/
LOCAL_DB_PORT=53306
LOCAL_HTTP_PORT=80
DJANGO_SUPERUSER_PASSWORD = password123
DJANGO_SUPERUSER_USERNAME = admin123
DJANGO_SUPERUSER_EMAIL = test@gamil.com

To get a minimum working .env file simply copy and rename the docker/public.env From the base YSE_PZ directory run

cp docker/public.env docker/.env

Running the docker containers

To run the docker containers, in the YSE_PZ/docker/ directory run

docker compose up

To bring the docker container stack down, in the YSE_PZ/docker/ directory run

docker compose down

Static files

To get YSE_PZ to see all the statics file run the following:

docker exec -it ysepz_web_container bash

Then in the docker container run,

python3 manage.py collectstatic

and type yes if asked about overwriting existing static files. To exit the docker container, run

exit

Adding a superuser

In order to be able to log into YSE_PZ you need to create a superuser. To do this, run the following command,

docker exec -it ysepz_web_container bash -c 'python3 manage.py createsuperuser --noinput'

This command will create a superuser with the username and password defined in the .env file.

Viewing webpages

Whilst the docker container stack is running go to http://0.0.0.0/login/ in your web browser and YSE_PZ should be running.

Native

Warning

The following native install instructions are here for archival purposes only. We strongly discourage you from using these instructions.

This is a easy guide to performing a local installation of YSE_PZ.

MySQL

If you don’t already have MySQL, it’s easy to get it using Homebrew. To get homebrew, run:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Then to install and start MySQL:

brew install mysql

Check your MySQL version with:

mysql --version

If you’re running version 5, you should try to upgrade to MySQL version 8 if possible:

brew upgrade mysql

After that, start the MySQL server with:

brew tap homebrew/services
brew services start mysql

Then set up your root password:

mysqladmin -u root password 'yourpassword'

Make sure to put single quotes around your password!

Anaconda Python

http://anaconda.org

Database File and settings.ini file

If you’re a YSE or UCSC team member (or other collaborator), ask D. Jones for these files. Otherwise, you can use the public versions of the YSE_PZ database (updated approximately whenever I feel like it, or please ask for the latest and greatest), which are YSE_PZ/public_settings.ini for the settings file and follow this Dropbox link for the database.

If you don’t need to ingest any new transients into the database, you can copy this file over to YSE_PZ/settings.ini and you’re good to go! Otherwise, still copy the file over and then see the Setting up the YSE-PZ Crons to finish the setup.

Installing the YSE_PZ Code

Should be straightforward:

git clone https://github.com/davecoulter/YSE_PZ.git
cd YSE_PZ
conda env create -f yse_pz.yml
conda activate yse_pz
pip install -r requirements.txt

Put the settings.ini file in the YSE_PZ/ directory (not the main repository directory, the directory with the same name one level down).

Please note that sometimes the extinction module is buggy. It is needed for some functionality but I would recommend trying to install it last.

Alternatively, I haven’t tried this myself, but - the latest YSE_PZ conda environment from my mac is included as yse_pz_latest.yml, so to avoid pip as much as possible you can try:

conda env create -f yse_pz_latest.yml
conda activate yse_pz

Starting the Web Server

In the YSE_PZ directory, run:

python manage.py migrate
python manage.py runserver

Then in a web browser on your computer, go to the url http://127.0.0.1:8000. If you’re initializing from the public database, the only existing user is admin and the password is set to changeme. Users are easy to create from the http://127.0.0.1:8000/admin page. After that, you should be good to go!

Setting up the YSE-PZ Crons

Running YSE-PZ as a living database that ingests new transient data requires setting up two crons to add new transients from the Transient Name Server and ingest new ZTF data from MARS. Ingesting new ZTF data for existing transients from MARS is relatively easy, while creating new transients from TNS requires setting up a TNS “bot” for yourself or your group at https://wis-tns.weizmann.ac.il/bots. See the TNS bulk reports manual at https://wis-tns.weizmann.ac.il/sites/default/files/api/TNS_bulk_reports_manual.pdf for more information. Once the API key has been set up, paste the key into the tnsapikey= line in the YSE_PZ/settings.ini file.

There are sometimes errors related to ingesting new transients, and for this reason it helps to link an email address to an account where error messages can be sent. This is the dbemail key in the YSE_PZ/settings.ini file. Unfortunately, to send those emails, you need to link an email account to YSE_PZ. You can do this by setting SMTP_LOGIN and SMTP_PASSWORD in the SMTP_provider section of the settings.ini file to your email username and password (you’ll need to change SMTP_HOST if the account is not gmail). You’ll also have to allow access to less secure apps in google, which is easy to find instructions for online.

Once all this has been set up, you can run the TNS cron with:

python manage.py runcrons YSE_App.data_ingest.TNS_uploads.TNS_recent --force

Every time you run this command (manually or with a cron, e.g., https://www.man7.org/linux/man-pages/man5/crontab.5.html), this will grab transients uploaded to TNS or updated within the last day. To change the time interval (in YSE, we do this every three minutes), change the tns_recent_ndays parameter in the settings.ini file.

Finally, to get new ZTF or TNS photometry/spectra for objects, the following cron will grab everything for transients with the statuses Watch, Interesting:, FollowupRequested, Following, or FollowupFinished:

python manage.py runcrons YSE_App.data_ingest.TNS_uploads.TNS_updates --force

For everything marked as Ignore for transients modified in the last 30 days, run this one:

python manage.py runcrons YSE_App.data_ingest.TNS_uploads.TNS_Ignore_updates --force

This can be a ton of transients, so it’s best not to run this too often.

Bugs, etc.

Please feel free to use the GitHub page for bugs/issues. Good luck!