IRAF Newsletter -- Number 14 -- April 1998

[ Previous ] [ Next ] [ Table of Contents ] [ Search this issue ]

The NOAO Mosaic Data Handling System

A number of observatories, including NOAO, are building wide field "mosaic" cameras. These cameras produce very large digital images using the technique of placing smaller detectors as close together as possible; i.e., a mosaic of detectors. This method overcomes limitations on the physical size of current digital detectors, such as CCDs, for covering large fields, and also decreases the readout time since all the CCDs can be readout in parallel. These new cameras provide many challenges for a data handling system.

The challenges faced by a data handling system include the following. By design these wide-field cameras produce very large images. This has implications for efficient data readout, transmission, storage, display, and processing. The readout from multiple detectors needs to proceed in parallel, hence pixels from the various CCDs are interleaved, and the data handling system must "unscramble" and otherwise decode (e.g., flip) the data stream coming from the data acquisition system. The mosaic nature of the data adds challenges in how to align, register, and combine the various pieces including dealing with the gaps between detectors.

NOAO, through the IRAF Group, has been developing a data handling system for CCD mosaic cameras. This Mosaic Data Handling System (MDHS) is used with the NOAO 8K x 8K CCD Mosaic Camera but it is not tied to this camera. The architecture of this system is open and flexible. The basic concept is that data and events are distributed between various "components" through a messaging system called a "message bus".

During a readout the data acquisition system opens a connection to the data handling system and sends a stream of messages indicating the start and end of the readout, header information, and interleaved data packets. One component of the MDHS, called the "data capture agent" (DCA), listens for this information and collects and unscrambles the data and header information to produce the output Mosaic observation file. This file, which is more accurately a type of live data structure on the message bus, is basically a collection of images from each detector (or from each amplifier if a CCD is read out with with multiple amplifiers). The data structure can be shared during or after readout with other components of the MDHS via the message bus and written to disk.

The disk format is a multi-extension FITS file. The IRAF FITS image kernel, described elsewhere in this Newsletter, allows users to directly interact with mosaic data from the MDHS. An IRAF package, called mscred , is being developed to process mosaic data stored in this format. Mscred processes the data from a collection of raw CCD images to final flux and astrometrically corrected images. The package provides tools for the important and common operation of removing the gaps in the mosaic data by taking several exposures with small shifts, called a dither sequence, and stacking the observations into a final single wide-field image with no gaps.

While operations may be performed directly on the disk format, greater efficiency is obtained by having specialized components access the data structure produced by the data capture agent through the message bus in real-time. One such component is a real-time display (RTD). The RTD displays the data in real-time doing some basic gain calibrations. The RTD includes specialized quick-look analysis features using plug-in modules. Other components, including IRAF tasks, can access and interact with the image data in the RTD.

Another MDHS component on the message bus is a data reduction agent (DRA). This component responds to events such as the end of a readout. Through a "recipe" the DRA can automatically process or reprocess data, using the latest calibration files. It is much like the traditional pipeline component of some telescope systems but it also provides a user interface (a GUI) to allow greater control by the observer over the data reduction process than is typical for a pipeline.

The MDHS is a long-term project. The most essential parts of the system described here have been implemented and have been in use with the NOAO CCD Mosaic Camera on Kitt Peak since the fall of 1997. This includes the message bus, the data capture agent, and the IRAF data reduction package, and an early version of the real-time display. The full real-time display and automated data reduction pipeline and DRA are still under development.

Doug Tody, Frank Valdes

There are a variety of documents on the NOAO Mosaic Data Handling System which provide much greater detail than we present here. These are referenced below. They are available on the Web from the address

Tody and Valdes, 1998, "The NOAO Mosaic Data Handling System":the most current and most detailed description of the MDHS based on a paper to appear in the Proceedings of SPIE Vol. 3355.
Valdes, 1998, "Guide to the NOAO Mosaic Data Handling Software":a user's guide for the IRAF MSCRED package.
Valdes, 1996, "Mosaic data structures": detailed design document for the Mosaic data format and keywords.
Valdes, 1996, "Mosaic reductions": initial design for the data reduction software.

The following are papers in the proceedings of the Astronomical Data Analysis Software and Systems (ADASS) conferences.

[ Previous ] [Next ] [ Table of Contents ] [ Search this issue ]

IRAF Group, National Optical Astronomy Observatories, P.O. Box 26732, Tucson, AZ 85726, Phone: (520) 318-8160, FAX: (520) 318-8360, Email

Posted: 07May1998