Data Ops Procedures
Table Of Contents
The SIPS Interface
Overview
The SIPS is designed to accept input data from LDAAC. Process it into higher MOPITT products and return these products to LDAAC for archive and distribution. The scheme is summarized in the following graphic:
.
Pickup
The MOPITT SCF expects data products delivered from LDAAC.These data are pushed via ftp to a designated directory. Currently, the directory is /mop1tt/ftp/pub/incoming/Pickup/.The expected files are summarized in the Expected files table.Expected files from LDAAC
| Name | Size (MB) | Frequency |
|---|---|---|
| MOP00SCI | 21 | 2-hourly |
| MOP00ENG | 2.8 | 2-hourly |
| MOP00TBL | 0.2 | 2-hourly |
| DLLAXMIS | 46 | daily |
| DLLAPMIS | 300 | daily |
| MOPEPHN0 | 0.5 | 2-hourly |
| MOPATTNF | 0.5 | 2-hourly |
Overflow
Since the anonymous ftp space has limited capacity, files are moved from the pickup area into an overflow area on the RAID. This directory was set to /MOPITT/EBS/MOSS-3/Overflow/.This move is accomplished by the overflowL0 script.
The Overflow dir has three sub-dirs:
- unmatched
- This dir is reserved for files whose name does not match any expected pattern of a delivered file.
- complete
- This dir is reserved for files which are recognized and for which we have received a DN of the correct size (implying complete transfer).
- incomplete
- This dir is reserved for recognized files whose size does not match the specified size in the DN. This tends to indicate an error in the DN generation rather than an incomplete transfer, but that bug may someday be fixed.
Archive Level 0
The archiveL0 script collects files from the specified overflow dirs and moves them into the Archive dir structure. The Archive dir has four sub-dirs:- DAS
- L0
- L1
- L2
The files are renamed as they are moved into the appropriate archive dir. The old filename and the new filename are recorded in a file called filenames.tbl.
Email Parser
Much of the communication between LDAAC and the SIPS is via email. The ICD specifies certain email messages to expect. The email parser will accept these messages and respond automatically.
| Name | When Sent | Info | Action |
|---|---|---|---|
| Subscription Notice (SN) | After a product has been ingested at the DAAC. | UR and data date | None |
| Distribution Notice (DN) | After a file is ftp pushed | UR, Local graunle ID, File size | compare reported file size with actual size. If they match, move the file to the Overflow/complete/ dir, otherwise move the file to Overflow/incomplete/. |
| PAN* | After files are inserted into ECS | Whether ingest/insertion was successful or not. | Successful PAN cause the removal of the delivery dir and PDR. Unsuccessful PANs do nothing until investigated. |
| PDRD* | If the PDR has an error | explanation of PDR problem | correct PDR (operator intervention) |
The L1 Processor
Stage L1
The script stageL1.txt assesses the contents of the archive. If needed files to run the L1 processor are not found, the script returns the list of missing files. Otherwise, the script copies the needed files into the staging dir and creates a PCF. The needed files to run the L1 processor are presented in the Staged Files for L1 Processing table.Staged Files for L1 Processing
| Name | Size (MB) | Number of Granules | Number of Files |
|---|---|---|---|
| Attitude | 0.5 | 12 | 12 |
| Ephemeris | 0.5 | 12 | 12 |
| MOP00SCI | 21 | 12 | 24 |
| MOP00TBL | 0.2 | 12 | 24 |
| MOP00ENG | 2.8 | 12 | 24 |
| MOPCH | 0.1 | 24* | 24* |
Run L1
Use the script runL1 to run the processor. It should take about 8 minutes to run a full day's data.Upon completion Operational QA is performed and the metadata is updated.Archive L1
After the L1 processor is run, the files it created are left in the staging dir. The script archiveL1 is run, which scans the staging dir. The created files are written to archive dir. If there are any anomalies, they are reported and the staging dir is left in place. Otherwise, the staging dir is deleted.Deliver L1
After the files have been archived, they need to be placed on our ftp disk for LDAAC to retrieve. The mechanism for this is via PDR, as described in the ./ECS ICD.The script deliverL1 collects the created files from the archive, puts them on the ftp area and creates a PDR.The files handled by this script are summarized in the Level 1 Products table.Level 1 Products
| Name | Size (MB) |
|---|---|
| MOP01 | 80 |
| MOP01ES | 0.03 |
| MOP01QE | ~0 |
| MOPCH | 21 |
The L2 Processor
Stage L2
The script stageL2.txt assesses the contents of the archive. If needed files to run the L2 processor are not found, the script returns the list of missing files. Otherwise, the script copies the needed files into the staging dir and creates a PCF. The needed files to run the L2 processor are presented in the Staged Files for L2 Processing table.Staged Files for L2 Processing
| Name | Size (MB) | Number of Granules | Number of Files |
|---|---|---|---|
| MOP01 | 80 | 1 | 1 |
| DLLAPMIS | 300 | 2* | 2* |
| DLLAXMIS | 46 | 2* | 2* |
Run L2
Use the script runL2 to run the processor. It should take about 4 hours to run a full day's data.Upon completion Operational QA is performed and the metadata is updated.Archive L2
After the L2 processor is run, the files it created are left in the staging dir. The script archiveL2 is run, which scans the staging dir. The created files are written to archive dir. If there are any anomalies, they are reported and the staging dir is left in place. Otherwise, the staging dir is deleted.Deliver L2
After the files have been archived, they need to be placed on our ftp disk for LDAAC to retrieve. The mechanism for this is via PDR, as described in the ./ECS ICD.The script deliverL2 collects the created files from the archive, puts them on the ftp area and creates a PDR. The files handled by this script are summarized in the Level 2 Products table.Level 2 Products
| Name | Size (MB) |
|---|---|
| MOP02 | 90 |
| MOP02B | 2 |
| MOP02Q | 80 |
Configuration Management
Under the main operational dir will exist three controlled sub dirs:- L1/
- This dir maintains the source,config, and executable of the L1 processor. The version is maintained by dirs which are named /proc-N.n/ (where N is the major version number and n is the minor version number). Within each /proc-N.n/ dir there is a src, bin, and one or more cfg-i dirs. The src dir contains the source code of the processor, and a makefile. The bin dir contains an executable called L1Processor. The cfg-i dirs represent the various configurations or inputs, where i is the config version. When the processor is run, a version is specified as N.n.i to identify a Major.minor.config version.
- L2/
- This dir structure should be parallel to the L1 dir
- SIPS - containing several sub dirs
-
- bin/
- This dir contains the SIPS executables.
- browse/
- This dir contains the IDL programs needed to generate BROWSE products.
- lib/
- This dir contains the PERL functions used by the SIPS executables.
- txt/
- This dir contains all the usage documents.
Acronym List
| DAAC | Distributed Active Archive Center |
| DN | Distribution Notice |
| DS | Data Server |
| ECS | EOSDIS Core System |
| EOS | Earth Observing System |
| EOSDIS | Earth Observing System Data and Information System |
| ESDIS | Earth Science and Information System |
| ESDT | Earth Science Data Type |
| FTP | File Transfer Protocol |
| ICD | Interface Control Document |
| IV&V | Independent Verification & Validation |
| LaRC | Langley Research Center |
| MOPITT | Measurements Of Pollution In The Troposphere |
| NCAR | National Center for Atmospheric Research |
| PAN | Product Availability Notice |
| PCF | Process Control File |
| PDR | Product Delivery Record |
| PDRD | Product Delivery Record Discrepancy |
| QA | Quality Assurance |
| SCF | Science Computing Facility |
| SN | Subscription Notice |
| SIPS | Science Investigator-Led Processing Systems |
| UR | Universal Record |
Daniel Ziskin