Pipeline Steps

This page tracks a rough mapping from the existing pipeline system to the new one. (Note that there are links to a password-protected page, http://snovae.in2p3.fr/snprod/doc/SNfCodeDoc.html that describes the existing pipeline steps in more detail.)

• Data transfer and summit cleaning

  Old: (Includes export_sync, hsi_import and "summit cleaning")

  New: we're pulling data from the CC rather than the summit, so we skip a lot of this. The scripts snf-pull-raw-data fetches a subset of data from CC, writing to the /project filesystem at NERSC. The scripts snf-archive-raw-images and snf-archive-raw-logs back that data up to the HPSS tape system.

• snf_header

  Old: This step writes every keyword of every FITS file header (matching a certain pattern) to a process_header table in a Postgres database (user/host defined in SnfDjangoMigr2.py). Table columns are added as needed by the FITS header.

  New: The snf-sync-db script synchronizes select keywords of every FITS file to a rawimages table in a SQLite3 database on /project/.../processing/db/images.db.

• snf_db_make

  Old: Meteo and FetchIAUC get external weather and supernova metadata and do some data munging.
  New: snf-pull-raw-weather-and-iauc fetches to /project/.../raw/[weather,iauc]; no data munging.

  Old: SnfUpdate reads log files for Run, Exposure, Pose information.

• snf_db_fill

• SyncTarget.py

• FlagRunKind.py

• plan_synthetic_arcs

• plan_file_quality

• plan_cube_generation

• plan_extract_star

• plan_multi_standard

• plan_photometric_ratios A

• plan_photometric_ratios B

• tabphotometricity

• plan_flux_solution

• plan_flux_calibration

• plan_gs_psf

• plan_cubefit

• plan_analyse_timeseries

• study_flux_quality

• update_idr_config

• build_idr