This document describes the ANDICAM data products delivered to observers via the SMARTS FTP repository at Yale. A separate FITS Header Specfication contains a detailed description of the ANDICAM FITS headers. Sample headers are available below.
For detailed, uptodate information about the CCD and IR detectors, see the ANDICAM Detector Characteristics page.
The raw data from each night are uploaded to a data server at Yale for further handling. Each active SMARTS project is assigned a Project ID code that is used to track all of their data. Some processing of the data are done either on-site at CTIO or at Yale before the data are delivered to the repository for pickup by the project investigators. These pre-distribution processing steps are detailed below for each detector channel.
The PI (or their designated principal contact) for each project will be contacted by SMARTS operations personnel at Yale to provide you with the login and password information for the Yale data respository. All data will be made available for download via ftp using password authentication; no anonymous ftp is available to protect the data from unauthorized access. For obvious reasons of security, none of the data server information (hostname, username, or password) will be provided on this website.
All data are provided in the form of gzip compressed FITS format images. Separate directories for each project hold CCD and IR images. The directories have names like:
OSU-03A-0001ccd OSU-03A-0001irwhere "ccd" and "ir" are self-explanatory, and the first part of the directory names are the SMARTS project ID for the program. Data for different projects are kept separately.
PIs are responsible for making sure that their data are retrieved in a timely fashion, and for notifying the Data Processing and Archiving Manager at Yale when their downloads are complete so the ftp disk space can be cleared for the next round of data.
Automatic Data Notification
Starting 2003 December 1, all PIs (or their project's designated principal contacts) will receive an automated email notification when data for their projects has been acquired by the ANDICAM. Unless there are unforeseen difficulties, most data should appear in the Yale ftp repository by noon (Eastern Time) on the day after they were acquired at CTIO.
The data files listed in the autonotification email messages are based on the raw observing logs created by the data-acquisition system during observing. These logs contain an unexpurgated listing of all data taken during the night, including junk frames and bad images that were subsequently deleted and re-observed (e.g., due to tracking/guiding failure, started by stopped prematurely by clouds, etc). The rare observer mistakes also enter the logs (e.g., right target but wrong obs file, etc.).
All such raw data issues are corrected during pipeline processing at Yale, so you will occasionally find a slightly different set of files waiting for you on the SMARTS ftp site. The ftp site contents should be considered definitive, whereas the autonotification email is primarily informational in nature, although it will also be a perfect match to the data waiting for you at Yale most of the time.
No pre-processing of CCD images is done at CTIO. Instead images are logged and then archived for uploading to Yale in the morning after the observing is done for the night.
Raw CCD data are gathered each morning by the Yale SMARTS team and sent through a basic 2-D data reduction pipeline at Yale that uses the NOAO IRAF package and the ccdproc task. This pipeline performs the following operations:
rccdYYMMDD.nnnn.fitsThe r is prepended to the filename to note that the images are "reduced", otherwise the rest of the filename is identical to the raw image filename that appears in the Nightly Observing Logs (this filename is also stored as the FILENAME keyword in the image FITS headers).
The CCD images are not trimmed by the pipeline. This results in a few very large (positive and negative) pixels that should be trimmed from the images by the user. The CCD trim box (in IRAF notation) is:
[17:1039,2:1025]Occasionally some bad pixels spill over into the images (the inevitable edge effects), but this trim box should cover most uses. This trim box will yield a 1023x1024 pixel final image ready for analysis.
IR data are pre-processed on-site at CTIO using an IRAF script that bins the images 2x2 in software with a bad pixel map. This procedure is described in detail on the ANDICAM IR Array Binning page. The images are processed on-site to reduce their size before uploading them to the Yale repository.
The resulting binned IR images are named
binir040408.0019.fitswhere bin has been prepended to the original raw-image filename to note that the images have been binned, otherwise the rest of the filename is identical to the raw image filename that appears in the Nightly Observing Logs (this filename is also stored as the FILENAME keyword in the image FITS headers).
Unlike the CCD images, no further pipeline processing (e.g., zero or flat-field correction) is performed on the images after they are uploaded to the Yale Repository. Individual investigators should decide how best to handle their IR data for further processing. See the Calibration Data for information on the availablility of IR flat fields and dark images.
The original, unbinned images are retained and stored on DAT tape. It is possible to arrange to receive the unbinned data by contacting the Data Processing and Archiving Manager. There will be a considerable delay in receiving your data, however, because the tapes must physically arrive from CTIO before the data can be made available.
Dome flats are acquired each week in the VRI and JHK filters, while sky flats are acquired (weather permitting) for the B filter. These data provide the basis for flat-fielding the CCD and IR images.
binir040414.domeK1.fits binir040414.domeK2.fits binir040414.domeK3.fits binir040414.domeK4.fits binir040414.domeK5.fits binir040414.domeK6.fits binir040414.domeK7.fitswhere "bin" notes that the frames have been binned following the usual procedures for IR images, 040414 is the observation date (here 2004 April 14), and the rest of the filename gives the light source (dome), the filter band (K), and the dither position (1-7). Y sky flats are acquired as weather permits using an analogous dome open/closed procedure to measure the parasitic thermal component.
A "dome" flat is a combination of flats taken with the flat-field screen's lamp on and off. The lights-off flat giving a measure of the thermal "parasitic" background due to the telescope, dome, etc., and lights-on contains both the illuminated screen and the parasitic components. The "dome" flats above have the parasitic component subtracted (i.e., Lights-On minus Lights-Off) for an average of (usually) 10 frames. A separate flat is acquired at each of the 7 dither positions, with a dither throw of 40. It is recommended that you select a dither scale of 40 (the default) to better match the flat fields.
IR flats should be stable on multi-week timescales, provided that the instrument has not been warmed up. If you wish to use dome flats for your IR data, find the flat fields closest in time to your data, usually bracketting your observations.
In general, ANDICAM is designed for synoptic observations of targets requiring only relative photometric calibration. This allows it to operate efficiently on both photometric and non-photometric nights, and regular photometric calibrations were not planned as part of the operational profile of the system.
Nonetheless, a few photometric programs have been approved, and even among the nominally synoptic programs there is a need for at least a zero-point reference to put their photometry on approximately the correct scale. As such, observations of a few selected CCD and IR photometric standard star fields are acquired on the occasional photometric night to at least provide zero-color photometric zero points. Programs requiring high-precision transformations will have to arrange to schedule additional calibration observations (e.g., covering a range of airmass and/or stellar colors) that will be charged against their time allocations at a rate determined by the SMARTS consortium board. Contact the SMARTS general manager (Charles Bailyn) for information.
At present, a general database of photometric calibrations is not available, and users who require photometric calibration should contact the Yale SMARTS personnel for information on obtaining and reducing standard star data. In the general community spirit of SMARTS, however, we would like to solicit contributions from users who have reduced their photometric standard star data and derived color and extinction terms for the ANDICAM CCD and IR channels. This will allow us to create a calibration database. Otherwise, we have no personnel free to do this for the consortium at the present time.
The following links are to annotated sample FITS headers for CCD and IR images, respectively. A detailed description of the FITS headers is given in a separate ANDICAM FITS Header Specification document.
Sample CCD image FITS Header
Sample IR image FITS Header
Updated: 2006 May 11 [rwp/osu]