SMARTS Consortium
CTIO 1.3m Telescope

ANDICAM Data Products

Updated: 2004 July 15

This document describes the ANDICAM data products delivered to observers via the SMARTS FTP repository at Yale. A separate FITS Header Specfication contains a detailed description of the ANDICAM FITS headers. Sample headers are available below.

Contents:

Yale SMARTS Data Repository
CCD Data
IR Data
Calibration Data
Sample FITS Headers

For detailed, uptodate information about the CCD and IR detectors, see the ANDICAM Detector Characteristics page.

Yale SMARTS Data Repository

The raw data from each night are uploaded to a data server at Yale for further handling. Each active SMARTS project is assigned a Project ID code that is used to track all of their data. Some processing of the data are done either on-site at CTIO or at Yale before the data are delivered to the repository for pickup by the project investigators. These pre-distribution processing steps are detailed below for each detector channel.

Network Access

The PI (or their designated principal contact) for each project will be contacted by SMARTS operations personnel at Yale to provide you with the login and password information for the Yale data respository. All data will be made available for download via ftp using password authentication; no anonymous ftp is available to protect the data from unauthorized access. For obvious reasons of security, none of the data server information (hostname, username, or password) will be provided on this website.

Data Format

All data are provided in the form of gzip compressed FITS format images. Separate directories for each project hold CCD and IR images. The directories have names like: 

   OSU-03A-0001ccd
   OSU-03A-0001ir
where "ccd" and "ir" are self-explanatory, and the first part of the directory names are the SMARTS project ID for the program. Data for different projects are kept separately.

Data Management

PIs are responsible for making sure that their data are retrieved in a timely fashion, and for notifying the Data Processing and Archiving Manager at Yale when their downloads are complete so the ftp disk space can be cleared for the next round of data.

Automatic Data Notification

Starting 2003 December 1, all PIs (or their project's designated principal contacts) will receive an automated email notification when data for their projects has been acquired by the ANDICAM. Unless there are unforeseen difficulties, most data should appear in the Yale ftp repository by noon (Eastern Time) on the day after they were acquired at CTIO.

Important Note:

The data files listed in the autonotification email messages are based on the raw observing logs created by the data-acquisition system during observing. These logs contain an unexpurgated listing of all data taken during the night, including junk frames and bad images that were subsequently deleted and re-observed (e.g., due to tracking/guiding failure, started by stopped prematurely by clouds, etc). The rare observer mistakes also enter the logs (e.g., right target but wrong obs file, etc.).

All such raw data issues are corrected during pipeline processing at Yale, so you will occasionally find a slightly different set of files waiting for you on the SMARTS ftp site. The ftp site contents should be considered definitive, whereas the autonotification email is primarily informational in nature, although it will also be a perfect match to the data waiting for you at Yale most of the time.

CCD Data

Pre-processing

No pre-processing of CCD images is done at CTIO. Instead images are logged and then archived for uploading to Yale in the morning after the observing is done for the night.

Pipeline Processing

Raw CCD data are gathered each morning by the Yale SMARTS team and sent through a basic 2-D data reduction pipeline at Yale that uses the NOAO IRAF package and the ccdproc task. This pipeline performs the following operations:

  1. Subtraction of overscan bias using data in the detector overscan columns (OVERSCAN task)
  2. Subtracts a Zero (aka "2D bias") frame from the image (ZEROCOR task)
  3. Divides the image by a normalized dome flat (FLATCOR task)
The resulting data are stored as 4-byte floating-point FITS data (BITPIX=-32) in files named 
   rccdYYMMDD.nnnn.fits
The r is prepended to the filename to note that the images are "reduced", otherwise the rest of the filename is identical to the raw image filename that appears in the Nightly Observing Logs (this filename is also stored as the FILENAME keyword in the image FITS headers).

Notes

The CCD images are not trimmed by the pipeline. This results in a few very large (positive and negative) pixels that should be trimmed from the images by the user. The CCD trim box (in IRAF notation) is: 

    [17:1039,2:1025]
Occasionally some bad pixels spill over into the images (the inevitable edge effects), but this trim box should cover most uses. This trim box will yield a 1023x1024 pixel final image ready for analysis.

IR Data

Pre-processing

IR data are pre-processed on-site at CTIO using an IRAF script that bins the images 2x2 in software with a bad pixel map. This procedure is described in detail on the ANDICAM IR Array Binning page. The images are processed on-site to reduce their size before uploading them to the Yale repository.

The resulting binned IR images are named 

   binir040408.0019.fits
where bin has been prepended to the original raw-image filename to note that the images have been binned, otherwise the rest of the filename is identical to the raw image filename that appears in the Nightly Observing Logs (this filename is also stored as the FILENAME keyword in the image FITS headers).

Pipeline Processing

Unlike the CCD images, no further pipeline processing (e.g., zero or flat-field correction) is performed on the images after they are uploaded to the Yale Repository. Individual investigators should decide how best to handle their IR data for further processing. See the Calibration Data for information on the availablility of IR flat fields and dark images.

Notes

The original, unbinned images are retained and stored on DAT tape. It is possible to arrange to receive the unbinned data by contacting the Data Processing and Archiving Manager. There will be a considerable delay in receiving your data, however, because the tapes must physically arrive from CTIO before the data can be made available.

Calibration Data

Flat Fields

Dome flats are acquired each night in the VRI and JHK filters, while sky flats are acquired (weather permitting) for UB and Y filters. These data provide the basis for flat-fielding the CCD and IR images.

CCD Flats
All CCD images are now automatically flat-field calibrated by the CCD pipeline. CCD flat fields images for data taken after 2003 August are available by special request to the Data Processing and Archiving Manager.

IR Flats
IR images are not flat fielded automatically. The dome flats at JHK or the Y sky flats, these are available from the Yale ftp respository in the "calibsir" directory. There are seven (7) sets of dome flats for each filter, with names like 
   binir040414.domeK1.fits
   binir040414.domeK2.fits
   binir040414.domeK3.fits
   binir040414.domeK4.fits
   binir040414.domeK5.fits
   binir040414.domeK6.fits
   binir040414.domeK7.fits
where "bin" notes that the frames have been binned following the usual procedures for IR images, 040414 is the observation date (here 2004 April 14), and the rest of the filename gives the light source (dome), the filter band (K), and the dither position (1-7). Y sky flats are acquired as weather permits using an analogous dome open/closed procedure to measure the parasitic thermal component.

A "dome" flat is a combination of flats taken with the flat-field screen's lamp on and off. The lights-off flat giving a measure of the thermal "parasitic" background due to the telescope, dome, etc., and lights-on contains both the illuminated screen and the parasitic components. The "dome" flats above have the parasitic component subtracted (i.e., Lights-On minus Lights-Off) for an average of (usually) 10 frames. A separate flat is acquired at each of the 7 dither positions, with a dither throw of 40. It is recommended that you select a dither scale of 40 (the default) to better match the flat fields.

IR flats should be stable on multi-week timescales, provided that the instrument has not been warmed up. If you wish to use dome flats for your IR data, find the flat fields closest in time to your data, usually bracketting your observations.

Bias Images

CCD Bias Images
2D bias (aka "zero") images are acquired for the CCD detector each night. These images are 0-second integrations obtained with the shutter closed, and should record only 2D bias signal present on the images due to the detector readout process. These images are combined into "zero" frames used by the CCD reduction pipeline for the ZEROCOR step.

IR "Bias" Images
The equivalent image for the IR array is called a "Dark" frame. This is something of a misnomer: the image is not a "dark frame" in the traditional sense, it in effect records the 2D "bias" information present in the IR images due to the 4-second reset/readout process. It is taken by inserting a cold "dark plug" filter into the IR camera beam (hence the name - the dark plug is a piece of aluminum occupying one of the IR filter wheel positions), and the IR array is reset and immediately read out many times. These images are available to investigators on the Yale repository as the "dark" frames that are stored in the same calibsir directory as the IR flat fields. Some people find them useful (i.e., the ones who requested we take them), but most people don't need to use them if they combine their dithered images together in the standard way to form a "self-sky image".

Photometric Standards

In general, ANDICAM is designed for synoptic observations of targets requiring only relative photometric calibration. This allows it to operate efficiently on both photometric and non-photometric nights, and regular photometric calibrations were not planned as part of the operational profile of the system.

Nonetheless, a few photometric programs have been approved, and even among the nominally synoptic programs there is a need for at least a zero-point reference to put their photometry on approximately the correct scale. As such, observations of a few selected CCD and IR photometric standard star fields are acquired on the occasional photometric night to at least provide zero-color photometric zero points. Programs requiring high-precision transformations will have to arrange to schedule additional calibration observations (e.g., covering a range of airmass and/or stellar colors) that will be charged against their time allocations at a rate determined by the SMARTS consortium board. Contact the SMARTS general manager (Charles Bailyn) for information.

At present, a general database of photometric calibrations is not available, and users who require photometric calibration should contact the Yale SMARTS personnel for information on obtaining and reducing standard star data. In the general community spirit of SMARTS, however, we would like to solicit contributions from users who have reduced their photometric standard star data and derived color and extinction terms for the ANDICAM CCD and IR channels. This will allow us to create a calibration database. Otherwise, we have no personnel free to do this for the consortium at the present time.

Sample FITS Headers

The following links are to annotated sample FITS headers for CCD and IR images, respectively. A detailed description of the FITS headers is given in a separate ANDICAM FITS Header Specification document.

CCD Images

Sample CCD image FITS Header

IR Images

Sample IR image FITS Header

Return to: ANDICAM Main Page

Updated: 2006 May 11 [rwp/osu]