MOPITT Operations (OPS) Procedures
The SIPS Interface
The SIPS is designed to accept input data from LDAAC. Process it into
higher MOPITT products and return these products to LDAAC for archive
and distribution. The scheme is summarized in the following graphic:
.
The MOPITT SCF expects data products delivered from LDAAC.These data are
pushed via ftp to a designated directory. Currently, the directory is
/mop1tt/ftp/pub/incoming/Pickup/.The expected files are summarized
in the Expected files table.
Expected files from LDAAC
| Name | Size (MB) | Frequency |
|---|
| MOP00SCI | 21 | 2-hourly |
| MOP00ENG | 2.8 | 2-hourly |
| MOP00TBL | 0.2 | 2-hourly |
| DLLAXMIS | 46 | daily |
| DLLAPMIS | 300 | daily |
| MOPEPHN0 | 0.5 | 2-hourly |
| MOPATTNF | 0.5 | 2-hourly |
Since the anonymous ftp space has limited capacity, files are
moved from the pickup area into an overflow area on the RAID. This
directory was set to /MOPITT/EBS/MOSS-3/Overflow/.This move
is accomplished by the overflowL0
script.
The Overflow dir has three sub-dirs:
- unmatched
-
This dir is reserved for files whose name does not match any
expected pattern of a delivered file.
- complete
-
This dir is reserved for files which are recognized and for which we
have received a DN of the correct size (implying complete transfer).
- incomplete
-
This dir is reserved for recognized files whose size does not match
the specified size in the DN. This tends to indicate an error in the
DN generation rather than an incomplete transfer, but that bug may
someday be fixed.
The archiveL0 script collects files from
the specified overflow dirs and moves them into the Archive dir structure.
The Archive dir has four sub-dirs:
The files are renamed as they are moved into the appropriate
archive dir. The old filename and the new filename are recorded in
a file called filenames.tbl.
Much of the communication between LDAAC and the SIPS is via
email. The ICD specifies certain email messages to expect. The
email parser will accept these messages and respond automatically.
| Name | When Sent | Info | Action |
|---|
| Subscription Notice (SN) |
After a product has been ingested at the DAAC. |
UR and data date |
None |
| Distribution Notice (DN) |
After a file is ftp pushed |
UR, Local graunle ID, File size |
compare reported file size with actual size. If they match, move
the file to the Overflow/complete/ dir, otherwise move the file to
Overflow/incomplete/. |
| PAN* |
After files are inserted into ECS |
Whether ingest/insertion was successful or not. |
Successful PAN cause the removal of the delivery dir and PDR.
Unsuccessful PANs do nothing until investigated. |
| PDRD* |
If the PDR has an error |
explanation of PDR problem |
correct PDR (operator intervention) |
*These types of email (PAN and PDRD) may be replaced by files
pushed to the SCF.
The script stageL1.txt assesses the
contents of the archive. If needed files to run the L1 processor
are not found, the script returns the list of missing
files. Otherwise, the script copies the needed files into the
staging dir and creates a PCF. The needed files to run the L1 processor
are presented in the Staged Files for L1 Processing table.
Staged Files for L1 Processing
| Name | Size (MB) | Number of Granules |
Number of Files |
|---|
| Attitude | 0.5 | 12 | 12 |
| Ephemeris | 0.5 | 12 | 12 |
| MOP00SCI | 21 | 12 | 24 |
| MOP00TBL | 0.2 | 12 | 24 |
| MOP00ENG | 2.8 | 12 | 24 |
| MOPCH | 0.1 | 24* | 24* |
* 4 optical trains x hot/warm/cold events x pre/post = 24. There could be
significantly fewer granules actually staged. 24 is the maximum.
Use the script runL1 to run the
processor. It should take about 8 minutes to run a full day's
data.Upon completion Operational QA is performed and the metadata
is updated.
After the L1 processor is run, the files it created are left in
the staging dir. The script archiveL1
is run, which scans the staging dir. The created files are written to
archive dir. If there are any anomalies, they are reported and the
staging dir is left in place. Otherwise, the staging dir is deleted.
After the files have been archived, they need to be placed on our
ftp disk for LDAAC to retrieve. The mechanism for this is via PDR,
as described in the SIPS/ECS ICD.The script
deliverL1 collects the created files
from the archive, puts them on the ftp area and creates a
PDR.The
files handled by this script are summarized in the Level 1
Products table.
Level 1 Products
| Name | Size (MB) |
|---|
| MOP01 | 80 |
| MOP01ES | 0.03 |
| MOP01QE | ~0 |
| MOPCH | 21 |
The script stageL2.txt assesses the
contents of the archive. If needed files to run the L2 processor
are not found, the script returns the list of missing
files. Otherwise, the script copies the needed files into the
staging dir and creates a PCF. The needed files to run the L2 processor
are presented in the Staged Files for L2 Processing table.
Staged Files for L2 Processing
| Name | Size (MB) | Number of Granules |
Number of Files |
|---|
| MOP01 | 80 | 1 | 1 |
| DLLAPMIS | 300 | 2* | 2* |
| DLLAXMIS | 46 | 2* | 2* |
*There may be two days of DAO data staged, but sometimes the processor will be
run with only one of each DAO file.
Use the script runL2 to run the
processor. It should take about 4 hours to run a full day's
data.Upon completion Operational QA is performed and the metadata
is updated.
After the L2 processor is run, the files it created are left in
the staging dir. The script archiveL2
is run, which scans the staging dir. The created files are written to
archive dir. If there are any anomalies, they are reported and the
staging dir is left in place. Otherwise, the staging dir is deleted.
After the files have been archived, they need to be placed on our
ftp disk for LDAAC to retrieve. The mechanism for this is via PDR,
as described in the SIPS/ECS ICD.The script
deliverL2 collects the created files
from the archive, puts them on the ftp area and creates a
PDR. The
files handled by this script are summarized in the Level 2
Products table.
Level 2 Products
| Name | Size (MB) |
|---|
| MOP02 | 90 |
| MOP02B | 2 |
| MOP02Q | 80 |
Under the main operational dir will exist three controlled sub dirs:
- L1/
- This dir maintains the source,config, and executable of the L1 processor.
The version is maintained by dirs which are named /proc-N.n/ (where N is the
major version number and n is the minor version number). Within each /proc-N.n/
dir there is a src, bin, and one or more cfg-i dirs. The src dir contains the
source code of the processor, and a makefile. The bin dir contains an
executable called L1Processor. The cfg-i dirs represent the various
configurations or inputs, where i is the config version. When the processor
is run, a version is specified as N.n.i to identify a Major.minor.config
version.
- L2/
- This dir structure should be parallel to the L1 dir
- SIPS/ - containing several sub dirs
-
- bin/
- This dir contains the SIPS executables.
- browse/
- This dir contains the IDL programs needed to generate BROWSE products.
- lib/
- This dir contains the PERL functions used by the SIPS executables.
- txt/
- This dir contains all the usage documents.
| DAAC | Distributed Active Archive Center |
| DN | Distribution Notice |
| DS | Data Server |
| ECS | EOSDIS Core System |
| EOS | Earth Observing System |
| EOSDIS | Earth Observing System Data and Information System |
| ESDIS | Earth Science and Information System |
| ESDT | Earth Science Data Type |
| FTP | File Transfer Protocol |
| ICD | Interface Control Document |
| IV&V | Independent Verification & Validation |
| LaRC | Langley Research Center |
| MOPITT | Measurements Of Pollution In The Troposphere |
| NCAR | National Center for Atmospheric Research |
| PAN | Product Availability Notice |
| PCF | Process Control File |
| PDR | Product Delivery Record |
| PDRD | Product Delivery Record Discrepancy |
| QA | Quality Assurance |
| SCF | Science Computing Facility |
| SN | Subscription Notice |
| SIPS | Science Investigator-Led Processing Systems |
| UR | Universal Record |
Dallas Masters
Last modified: Thu Oct 14 14:50:36 MDT 1999