NAME:
 phot2db
 PURPOSE:
 Process a file of external photometry and add to phot database.
 DESCRIPTION:
 Reads a file of external photometry data with format specified below. 
 It creates entries in two tables of the phot database, 
 reference and data. The file defines a REFID in its header section,
 which corresponds to an entry in phot.reference. Ordinarily this will
 be a new REFID- if not, the existing entry in phot.reference will be
 updated and existing entries in phot.data will be purged.
 CATEGORY:
 Photometry
 CALLING SEQUENCE:
 phot2db, fn
 INPUTS:
 fn   filename to be read and processed.
 OPTIONAL INPUT PARAMETERS:
 KEYWORD INPUT PARAMETERS:
 ADMIN     - String, if specified, interpreted as an email address (or list)
               to cc any email generated. If the contact information
               obtained from the data file is unavailable or bad,
               e-mail will addressed directly to ADMIN in the'to' field.
 DATABASE  - Name of database in which the reference and data tables reside,
               by default 'phot'.
 DATATABLE - Name of data table in data base, by default 'data'.
 NOCONTACT - Flag, if set, the contact information specified in the data
                file will not be used to send e-mail. The mail will be
                sent to ADMIN only. If NOCONTACT is specified without ADMIN
                it is equivalent to NOMAIL. Note that the contact field
                is always mandatory in the data file.
 NOMAIL    - Flag, if set, the normal email sent in case of errors and
               anomalies is suppressed. Errors will be printed to screen.
 PATH      - String, if specified, is a path that will be prepended to fn.
 REFTABLE  - Name of reference table in data base, by default 'reference'.
 ESQUELCH  - Maximum number of error lines in email messages, default 150.
                Note that the preamble and data base update summary is always
                printed regardless of ESQUELCH and that 'error lines' pertain
                to header and data sections of the file, as well as a listing
                of the header section of the file appended to the end. ESQUELCH
                must be greater than 0.
              
 TEST      --Flag, if set, no databases are modified, queries that would be
                generated are listed to the screen.
 OUTPUTS:
 KEYWORD OUTPUT PARAMETERS:
 COMMON BLOCKS:
 SIDE EFFECTS:
 Updates tables data and reference in the phot database. It normally sends
 email to the specified contact person (and admin) if errors or anomalies are
 found in the file read. 
 RESTRICTIONS:
 The header section of the file must specify at least REFID, OBSCODE, and
 CONTACT.
 The data section of the file must have at least 1 valid line, otherwise,
 no changes are made to the database. 
 The REFID must not begin with "LO-" which is reserved for Lowell Observatory
 internal use. 
 PROCEDURE:
 The program is normally run as part of a cron job on a directory of incoming
 photometry data. The data base tables are silently updated, unless errors or
 anomalies are found. These are reported through email only. Once processed,
 an external script moves data files to an accumulating archive.

 The format of the photometry data files (as of 2006 Nov 21)::

 The data file is composed of two sections, the header information and
 the data.  The header can be seen as an unbroken list of lines that have
 the form of an identifier followed by text.  It is generally expected
 that one file will contain all the data that might result from a single
 night of observation.  However, this is not required.  It is allowed
 for a file to contain more than one night of data in which case the
 RunDate information will not be provided.  It is NOT allowed to have
 multiple files for a single night on a single telescope and instrument
 that share a RefID.  The basic rule will be that one file = one RefID.
 Also, RefID must be unique among all possible datasets.
 Also, RefID must be unique among all possible datasets.
 The specimen file below includes the recognized header keywords.  
 The order is not important- however the DATA line must follow
 all the header lines.

 REFID.....:
 RUNDATE...:
 INSTRUMENT:
 OBSCODE...:
 TELESCOPE.:
 OBSERVER..:
 MEASURER..:
 CONTACT...:
 COMMENTS..:
 DATA:
   # JD RA DEC FILTER MAG ERR OBJNAME
   # 2443741.49403 12:12:12.1 +35:35:35 V 13.456 0.023 SAO 160688
   # 2443741.59403 x x R 14.567 0.031 SAO 120107
   # 2443741.69403 14:12:34.2 +17:23:12 R 14.567 0.031
   #




 The header contains keywords that are formatted as a name of up to
 10 characters in length and padded to a length of 10 by periods.
 The padded name is to be followed by a colon and then a space before
 the data it contains.  The order is not important.  Some keywords
 can usefully appear more than once providing for multi-line entries.
 These fields are collected in the order seen in the header.  It is an
 error for a non-multi-line keyword to appear more than once.  Note that a
 multi-line comment need not be contiguous in the header.  Blank (empty)
 lines and any line that starts with '#' are ignored no matter where
 they appear in the file.  The end of the header is noted with 'DATA:'
 (not padded).  After this line is seen no further header information is
 allowed and the rest of the file is expected to contain the data.

 The header keywords contain the following information:

 REFID

 This is a string, currently limited to a maximum of 20 characters.
 This string must be unique across all datasets.  The string really
 should be uniquely tied to a single file.  When the database is loaded
 from the file, the REFID value is used to identify old values that may
 have been previously loaded.  Prior to loading the new file, these old
 values will be deleted in their entireity and the new values will be
 placed into the database.  No attempt is made to match up values in
 the file with previously loaded data.  The actual construction of the
 REFID is not defined here but a good working example is to construct a
 string that looks like: XXX-YYMMDD-IIII where XXX are the initials of
 the contact (presumed to be responsible for the data), YYMMDD is the
 rundate YY last two digits of the year, MM the month, and DD the day
 in UT.  In cases where the UT date changes during the night choose the
 dominant UT date for the night.  For example, at CTIO, the data often
 changes an hour or so into the night.  In this case the UT date at local
 midnight is appropriate.  Actually, the UT date of local midnight is
 usually appropriate though keep in mind that local midnight does not mean
 "when is it midnight according to a local clock".  Finally, IIII is a
 string that identifies the instrument (does not need to be exactly four
 characters long).
 It is currently required that the REFID be a string with three non-null
 substrings separated by hyphen (- not _). For example,
 XXX-YYMMDD-IIII is legal but YYMMDD or XXX-IIII are not (nor is XXX--AAAA).
 This restriction is needed to avoid propagating RefID's that could conflict 
 with internal usage at Lowell Observatory.

 RUNDATE

 This field is not required but is very helpful in many cases.  This should
 be a 6-digit string, YYMMDD with the same meaning discussed under REFID.
 If not supplied this will be set to NULL in the database.  If the data
 being loaded apply to a single night this field makes a lot of sense.
 Data that may come from a larger collective source (SDSS photometry database
 for instance) the RUNDATE will be meaningless.
 for instance) the RUNDATE will be meaningless.
 INSTRUMENT
 INSTRUMENT
 This is a short identifying string, maximum of 12 characters.  There should
 be a one-to-one correspondence between the instrument being used and this
 name.  It is most useful if this really is a useful identifying string that
 is unique and constant for a given instrument.
 is unique and constant for a given instrument.
 OBSCODE
 OBSCODE
 This is the 3-character standard observatory code as defined by the
 Minor Planet center.
 Minor Planet center.
 TELESCOPE
 TELESCOPE
 Descriptive name of the telescope used to collect the data.  Maximum length
 saved in the database is 80 characters.
 saved in the database is 80 characters.
 OBSERVER
 OBSERVER
 [Multiple line field] Name(s) of the observers who collected the data at
 the telescope.  This field is not required but is strongly encouraged for
 normal nightly datasets.  You can put multiple names on one line or use
 one line per observer.
 one line per observer.
 MEASURER
 MEASURER
 [Multiple line field] Name(s) of the who processed the raw data and generated
 the numbers in this file.  Usually this will be one person but multiple names
 are allowed and encouraged if appropriate.
 are allowed and encouraged if appropriate.
 CONTACT
 CONTACT
 Email address of the primary contact regarding the data.  This should be the
 contact address of someone that can be contacted long after the data are
 reduced and posted in case of questions.  It is not appropriate to use a
 transient worker as the contact (eg., a summer student).  It is also assumed
 and normal to expect that the contact is mentioned by name as one of the
 measurers.
 measurers.
 COMMENTS
 COMMENTS
 [Multiple line field]  Any comments that should accompany the data would be
 placed in this field.  For that sake of formatting by any system that reads
 these commands and generates output, a blank comment line (keyword but no
 text) will be preserved and will indicate a paragraph break.  Line breaks
 within a "paragraph" may or may not be preserved.
 within a "paragraph" may or may not be preserved.
 DATA:
 DATA:
 Once this keyword is seen the file processing switches to data mode
 reading.  The data format is not really a fixed format file.  Instead,
 the lines are read by word tokens and there must be a minimum of 6
 "words" on every line.  Words are separated by one or more blanks.
 Leading and trailing blanks are ignored.  The first word is the Julian
 Date and an F13.5 format is recommended though more digits can be provided
 if desired.  The second word is the J2000 Right Ascension of the object
 in either sexigesimal format (HH:MM:SS.s) or decimal hours.  The third
 word is the Declination (DD:MM:SS or decimal degrees).  The RA and Dec
 should NOT be supplied if this is not a measured quantity.  In other
 words, do not provide catalog positions or ephemeris calculations.
 If you do not have measured positions, then put a single 'x' character
 in place of each of the two fields. The fourth word is the name of the filter.
 Blanks are NOT allowed in a filter name.  This name should be a standard
 name to the extent possible.  For instance, Johnson filters would be V
 and SDSS would use something like r'.  The filter string may not execeed 10
 characters in length. The next two words are the standard
 magnitude and its error.  Use as many digits of precision as appropriate.
 The remainder of the line is used for the object name.  Embedded blanks are
 allowed in the object names though multiple blanks will get collapsed to a
 single blank.  It is allowed (and normal) to omit the object name when a
 position is provided.  If the position is not provided then the object name
 is required.  Again, blank lines and lines starting with # are skipped when
 reading.


 MODIFICATION HISTORY:
 2006/11/21, Written by Peter L. Collins, Lowell Observatory