Name: nirspec_pipe_testing_tool
Owner: Space Telescope Science Institute
Description: null
Created: 2017-11-29 21:39:10.0
Updated: 2018-05-17 17:40:17.0
Pushed: 2018-05-17 17:40:16.0
Homepage: null
Size: 357
Language: Python
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
Simply put, a Pytest is a pass or fail Python test. For instance, with the WCS step, we have Python scripts (which we are calling auxiliary code within the frame of the testing tool) that compare the pipeline product with the ESA corresponding intermediary file, and calculates a difference. The Pytest is to assert if that difference is less than or equal to an X threshold value. Hence, a failed test means that the condition was not met. If an error should occur with the pipeline, the test will be flagged as an error.
NOTE: This guide assumes that Conda has been installed. If you have not yet done so, please follow the instructions at: https://astroconda.readthedocs.io/en/latest/ Please use python 3.5
THREE THINGS BEFORE STARTING
I.- You may want to clean your PYTHONPATH so that you do not get mysterious failures. To do this simply type the following command in the terminal:
t PYTHONPATH
You can do this every time you run the pytests, or when you start getting strange failures. Another option is not to define PYTHONPATH at all in the .profile (or equivalent: .bashrc, .bash_profile, .cshrc, .login, etc.) file.
II.- If you work outside the internal network, i.e. in the visitors network or at home, you also want to set the following environment variables in the terminal or add them to your .profile (or equivalent) file:
rt CRDS_SERVER_URL=https://jwst-crds.stsci.edu
rt CRDS_PATH=${HOME}/crds_cache
These changes will not affect your work while working with the internal network at ST.
III.- A brief description of what each pipeline step does, as well as a brief description of all the pytests implemented in the tool, the tests that are still in concept phase, and the tests that are still needed, can be found in the Confluence space for PTT. You can get to it from the main page of NIRSpec/NIRSpec JWST Pipeline/NIRSpec Calibration Pipeline Testing Tool (PTT), or by clicking in the following link: https://confluence.stsci.edu/pages/viewpage.action?pageId=123011558
a. Conda environment for this testing campaign:
a create -n jwst_b713 --file url_depending_on_your_system python=3.5
As bugfixes are announced your current pipeline software may be updated by issuing the command:
a update -n jwstdp-0.9.3rc1 --file http://ssb.stsci.edu/releases/jwstdp/0.9.3/latest-linux
NOTE: If you need to use the development version of the pipeline then do the following:
a create -n jwst_dev -c http://ssb.stsci.edu/conda-dev jwst python=3.5
Then, to update the development environment, activate the environment and then type:
a update --override-channels -c http://ssb.stsci.edu/conda-dev -c defaults --all
To get a specific version of the pipeline you need to do 2 steps: i) Find the version you want with (newest at the bottom)
a search -c http://ssb.stsci.edu/astroconda-dev jwst
ii) Install the version you want
a install --override-channels -c https://ssb.stsci.edu/astroconda-dev -c
defaults jwst=version_you_want
where `jwst=version_you_wantis usually something like ``jwst=1.2.3dev456```.
b. Configuration files corresponding to this build. Create a directory (e.g.
the following command within the conda environment you just created (see step 2).
collect_pipeline_cfgs .
ctivate the conda environment for testing the pipeline, e.g. type:
source activate jwst_b7.1
here on, every step of this guide should happen within the conda testig environment.
:
you forget what did you name your new environment type:
conda env list
will list all environments you have created.
you want to remove an old testing environment type:
conda env remove -n name_of_your_old_environment
nstall the following plugins for the pytests to work properly. Within the conda
ing environment, type:
pip install msgpack pip install pytest-html
: Every time you create a new conda environment you need to install these plugins.
lone the repository so you have the PTT. To do this click at the top right
his page, in the dropdown button that says clone or download, then copy the ulr that
ars there. Now, within the conda testing environment, go to or create the directory
e you want the PTT to "live" in. However, make sure that the configuration files
ctory is at least at the top level of the directory tree where the PTT will live, e.g.
```b7.1cfg_files``` directory and the ```nirspec_testing_tool``` directory can be at
same level, but the ```b7.1cfg_files``` directory cannot be inside the
will be picked up by Git and it will try to put them in your space. Remember you are in the GitHub repository page so go all the way to the top of the page, click on the green button and copy the ulr that appears there.
clone the_ulr_you_copied
After this is done, you should see a full copy of the PTT in your directory.
NOTE:
If you have already cloned the repository, in the terminal go to where you placed
the `nirspec_pipe_testing_tool` directory. Then, use the following command to update
the code:
If, however, you had written script(s) in the tool's directory tree, git will not let you continue until you move the script(s) to another directory.
Prepare the data to run through the pipeline. To do this:
a. Copy the test data you will use from the NIRSpec vault directory. Go to the directory where you want to have the testing data, and from there type:
r /grp/jwst/wit4/nirspec_vault/prelaunch_data/testing_sets/b7.1_pipeline_testing/
test_data_suite/the_data_you_want_to_copy .
b. In the directory where you copied the test data, you will need to run a script PER fits file you want to test. Do not worry, this will only be done once. This script will create a new subdirectory with the necessary input file to run the SSB script that converts raw data into uncal type files. You can choose to either keep this subdirectory, or tell the script to remove it after the operation is done. In the terminal type:
on /path_to_the_testing_tool/nirspec_pipe_testing_tool/utils/
prepare_data2run.py fits_file.fits MODE -u
where the MODE is expected to be one of: FS, MOS, IFU. This command will update the uncal
keyword header without creating a new file, and will also keep the subdirectory. To
remove it, simply add `-rmat the end. To save the keyword changes in a new fits
file (instead of updating), remove the ``-u```. The new uncal fits file is now ready
for pipeline ingest.
c. Optional. If you want to see the header of any file, you can use the another script
in the `utils` directory of the PTT. If you just want to see on-screen the
header, go where your fits file “lives” and type:
on /path_to_the_testing_tool/nirspec_pipe_testing_tool/utils/
read_hdr.py fits_file.fits
This command will show the main header. To save the header to a text file add a `-s
at the end. If you want to see/save a different extension add at the end ``-e=1``` for
extension 1, and so on.
d. Now, the data is ready to be ran through cal_detector1. Please go ahead with step 6 of this guide to do that.
Set the PTT configuration file. In a text file editor, you are going to modify the
configuration file named `PTT_config.cfg`, which lives at
is the file that controls all the input that the tool needs. Please open it and
sure that:
All the paths point to the right places. The files can be located anywhere, but both, the pipeline and the tool, will run faster if the files are local on your computer.
The input file for the PTT is the final output file from calwebb_detector1.
The adequate mode for the data to be tested is set correctly, choices are: FS, IFU, or MOS.
The steps that you want to be ran or not are set to True or False.
In the bottom part of the file, all the additional arguments for the PTT are correct, e.g. threshold values, figure switches, and additional fits files.
Run the `calwebb_detector1pipeline. The final output of this is the level 2 data
required to run the PTT. In a terminal, go into the directory where the testing tool lives
(i.e. at the level of the ``calwebb_spec2_pytests``` directory), and make sure that the
testing conda environment is on. Now type:
on ../utils/run_cal_detector1.py /path_where_the_uncal_file_lives/uncal_file.fits -sbs
This script will execute the calwebb detector 1 pipeline step by step. If you want to run
it in a single run, using the configuration file that you have for it in the `utils
directory, simply remove the ``-sbs``` from previous command.
If everything went well, you will see a text file called
ut fits file, and the time each step took to run. However, if you chose to run the
ebb detector1 in a single run, you will only see the total running time. This text
, along with the intermediary products will be located in the path you set for the
NOTE:
`change_filter_opaque2science,
which lives at ``/nirspec_pipetesting_tool/calwebb_spec2_pytests/auxiliary_code```.on ../calwebb_spec2_pytests/auxiliary_code/change_filter_opaque2science.py
/path_to_the_file.fits
If all went well and you have a gain_scale.fits file, you now are able to run the MESA calwebb detector 1 testing tool. Steps to obtain the MESA testing tool:
a. At the same level as the top directory of the PTT (i.e. the `nirspec_pipe_testing_tool
directory), create a new directory called ``MESA_cal_detector1```.
b. Inside `MESA_cal_detector1`, you will clone their Git repository. Please follow the
directions for this at:
http://calibration-pipeline-testing-tool.readthedocs.io/en/latest/
c. Now, in the `utilsdirectory of ``nirspec_pipe_testing_tool`, you will find a
sample json file that you can modify in order to use as input for the MESA calwebb
detector 1 testing tool. Notice that it differs from the example given in the MESA
documentation. In our example, the steps are in the order of the pipeline, to make it
easier to determine what file is input to which step. Please copy our sample json file
from theutils`` directory into the `MESA_cal_detector1` directory you created,
and or modify the json file with the name and path of your intermediary fits products
from running the calwebb_detector1 pipeline.
d. Run the MESA testing tool with the following command:
_pipeline --config ./cal_detector1_input.json
Please record your progress in our testing campaign Confluence page: https://confluence.stsci.edu/display/JWST/NIRSpec+Pipeline+Testing+Build+7.1+part+2.
Please remember to make a PDF export of the html report, so that it does not get corrupted when you move it to another directory.
Ready to run PTT. Go back to the directory where PTT lives and into the
working directory you indicated in the ```PTT_config.cfg``` file, and make sure
the input file for the PTT matches the file you just copied into the working
ctory. Now, to ensure that everything is in order, and to see what pytests will be
uted and in which order type:
pytest –collect-only
Do the first PTT run. As an output of the testing tool you will see an html
file, `report.html`, and an intermediary product file name map will appear in the
aved in the path you indicated at the ```PTT_config.cfg``` file with the variable
st -s --config_file=PTT_config.cfg --html=report.html --self-contained-html
The `-s` will capture all the print statements in the code on screen. If you want to
save the on-screen output into a text file simply do:
st -s --config_file=PTT_config.cfg --html=report.html --self-contained-html
> screen_output.txt
and this will create the `screen_output.txtfile in the ``calweb_spec2_pytests```
directory.
Report your findings. If all went well, you should have the html report in the
ucts that appeared in the working directory. Each section of the report specifies
tests passed or failed, and, if there were any errors, the report will tell you
h the name of the script where the error occurred) if it is a pipeline error or a
error. Keep updating your testing progress in our testing campaign Confluence page:
s://confluence.stsci.edu/display/JWST/NIRSpec+Pipeline+Testing+Build+7.1+part+2
Please follow these actions for reporting your progress, depending on the outcome:
The html file is now portable, i.e. it will not get corrupted if you move it to another directory.
Place the html file in the Confluence page. You can also create a PDF export from the report and use that file for the Confluence page and for any sharing purposes.
If there are no testing tool errors and some of the pytests failed, you can check
off the steps and please put that report in Table 2 of the Confluence page.
If there are testing tool errors please place your report in the Confluence page, add a small comment in the “Testing Tool” column of Table 2 in the Confluence page, and send an email to Maria Pena-Guerrero (pena@stsci.edu), so that the PTT error can be addressed as soon as possible.
If there are errors with the pipeline, please do the following: i) Look up if there are any JIRA issues already filed related to the error. ii) If there are JIRA issues open you can add a comment that you also encountered the problem for your data. iii) If there are closed JIRA issues you can re-open them and report in the comments section. iv) If there is no JIRA issue for the problem please go ahead and create one. You can find the NIRSpec guidelines for doing so at: https://confluence.stsci.edu/display/JWST/NIRSpec+Guidelines+for+Pipeline+Issue+Reporting and copy the link (or JIRA issue number) next to your report.
Keep updating the html report as you continue with the testing, and including the final report.
Put final results in the NIRSpec vault. At the end of the testing campaign, please do the following within your working directory:
a. Create a new directory called `yourname_MODE_Name_of_the_raw_data_file_results,
e.g. ``maria_IFU_NRSSMOS-MOD-G1M-17-5344175105_1_491_SE_2015-12-10T18h00m06_results`.
(If you already copied a previous version of your results and you want to keep it then add
a_v2`` -or the version number that corresponds- after the word results, e.g.
nside that new direcoty place all the intermediary fits products as well as the html
rt
lso inside that directory, create a text file named ```YourNameMODEresults.txt```,
```mariaIFU_NRSSMOS-MOD-G1M-17-5344175105_1_491_SE_2015-12-10T18h00m06_results.txt```.
his text file you will only type the full path where you obtained the testing data,
/grp/jwst/wit4/nirspec_vault/prelaunch_data/testing_sets/b7.1_pipeline_testing/
test_data_suite/IFU_CV3
lace the results directory you created in the staging directory of the
pec vault, and send Gray an email (gkanarek@stsci.edu). The path of the staging
ctory is:
e. Finally, change the permissions recursively on all files in your newly created directory, so that the NIRSpec curators team can move the data. To do this type the following command in a terminal:
O KEEP IN MIND
text file containing an intermediary product name map will be created in the pytests
ctory.
any of the central store directory calls do not respond (e.g. when looking at the
s), the pytest will be skipped even if the step is set to True in the config file.
failing message will say that the step was set to False (this is a known bug). To
e the tests run, you will have to download the files the tool is calling, and change
corresponding paths in the configuration file.
e output in the terminal can be a bit overwhelming if there was a failed test or an
r, since it shows both, the pipeline messages and the PTT messages. In the html
rt is much clearer to understand what happened.
part of the testing campaign, it is important that you run the pipeline from the
and line as well, and that you make sure that the outcome intermediary files are
istent with the ones ran with scripts, i.e. the PTT. This sanity check is minor
important to verify.
he ```utils``` directory there are two text files named
can use from the terminal to run the calwebb_detector1 and calwebb spec2 steps,
ectively.
nally, remember that:
henever you need to read either the main or science headers of a file,
can always use the ```read_hdr.py``` script located in the ```utils``` directory of
testing tool.
f you need to change/add a keyword value to a specific extension of a file, you can
the ```change_keywd.py``` script, also located at the ```utils``` directory.
OSSIBLE OUTCOMES OF THE PYTESTS
ssed = the assertion was true, so the test condition was met.
iled = the assertion was false, the test condition was NOT met (there will be an
rtionError on-screen and in the html file, with a clear PTT customized message of
happened).
ipped = the test was skipped (there will be a message on the html report and on-screen
aining why the test was skipped).
ror = this is a coding error, or a bug in either the pipeline or the PTT. Please
ow the appropriate steps described in the "For Testers" section of our testing
aign Confluence page:
s://confluence.stsci.edu/display/JWST/NIRSpec+Pipeline+Testing+Build+7.1+part+1
DDING TESTING ROUTINES
can add additional testing routines. We prefer if these are written in python 3.5,
ver, due to time constraints, we devised a method for including scripts in other
raming languages. For this testing campaign, the guidelines for auxiliary code are
following:
rograming language = preferably Python 3.5, however, other languages (e.g. IDL, C)
OK.
lease comment as much as you can so that it is easy for other testers to follow
code.
t the end of your script you must create a pass or fail test, e.g. after you calculate
ot-mean-square check if that value is within some range. If instead you have a
shold value, please make the threshold an input variable with a default value.
our script must produce a text file named ```scriptname_result.txt``` at
end, which will only contain one word: True or False.
reate a directory named ```scriptname``` with the following contents:
text file named ```scriptname_init.txt```, and in there type in line 1
pipeline step after which your code should be run, in line 2 the language you used,
in line 3 the exact command you would use to run the script. If this is an IDL script,
2 should be the exact call from within the IDL session.
he script you wrote.
ne level up from where you created the script directory, create a new directory named
This text file will only contain the following path:
ow, copy the ```YourName_PTT_auxiliary_code``` directory in the NIRSpec vault, and
se send an email to Gray Kanarek (gkanarek@stsci.edu). The vault's path is:
If you have any question of what a specific step does, you can get a description at: http://jwst-pipeline.readthedocs.io/en/latest/jwst/pipeline/description.html#stage-2-spectroscopic-pipeline-step-flow-calwebb-spec2