BIAD/ctrp_datawarehouse_bddtests

Name: ctrp_datawarehouse_bddtests

Owner: NCI CBIIT Development Sandbox

Owner: Biomedical Informatics Applications Development

Description: For BDD tests for the Data Modernization Project

Created: 2017-07-24 18:44:45.0

Updated: 2018-01-22 19:28:06.0

Pushed: 2017-08-09 18:20:37.0

Homepage: null

Size: 412

Language: Ruby

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

ctrp_datawarehouse_bddtests

For BDD tests for the Data Modernization Project

How to prepare BDD test for CTRP Data warehouse CTRP Data warehouse bdd tests are based on CTRP DW pipeline scripts You need to create one feature per script

1. create cucumber feature file
2. you need to create one feature file for one CTRP DW pipeline script
3. just make a copy feature/Cucumber/DW_Organization.feature because all feature should have same structure
4. in every feature file there actually only has three tests
   • the first one is the test to make sure the target DW table is there and all columns are correct and the “key” column should have exact same values comparing with seed data
   • the second one is the test to make sure every aspect of pipeline logic is executed correctly, the picked records should have exact values that is prepared in seed data
   • the third one use a validator class to valid randomly picked records
     
5. create seed data for the first test in the feature file for the first test, we need to check one “KEY” column has same values compare with seed data, we need to pick one column (normall it is the key column if there is one), then according the script to calculate how to get the values from input database (PO, PA, GRANTS), then use PgSeedData methods to create a line in seed_data_setup/generate_dynamic_seed_data.rb then everytime before the cucumber tests get triggered, seed_data_setup/generate_dynamic_seed_data.rb need to be executed first (it's written in travis.yml) to generate the seed data
6. create seed data for the second test in the feature file
   • in the test scenario, you need to cover all columns with different possible values, please check the data pipeline script logic
   • then use the scenarios to build all related PO, PA and GRANTS records.
   • you need to write those records in seed_data_setup/seed_stories/PO(PA, GRANTS) folder, there are related csv files for every table, if you cannot find csv, use PgSeedData.create_seed_story generate it
   • after you finish the story csv, run seed_data_setup/generate_local_seed_dump.rb, it will automatically download seed dump from s3, load into local postgres, push the new data to it, re-dump it and upload to s3
   • in the whole workflow, the seed dump will be used to generate PO PA and GRANTS table
7. create validator for the third test in the feature file
   • you need to look at the DW pipeline script for this table first, know about all logic in it
   • then create a new validator class in feature/support/validators name it as _validator.rb
   • you can look at the organization_validator.rb, all validator class should be subclass of CtrpDwBaseValidator
   • there must have two methods get implemented: target_table and validate
   • target_table: you need to put the exact Data Warehouse table name as the return value
   • validate: this should take a hash (the single record that need to be checked), put all database query here to verify the data
   • it's recommended to use HelperMethods.actual_eq_expect the compare values

Environment variables that need to be setup in both local and INT tier: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY

BDD_BOT_SLACK_TOKEN (please ask Vivek)

CTRP_DB_HOST CTRP_DB_PORT

CTRP_DB_NAME_GRANTS CTRP_DB_USERNAME_GRANTS CTRP_DB_PASSWORD_GRANTS

CTRP_DB_NAME_PA CTRP_DB_USERNAME_PA CTRP_DB_PASSWORD_PA

CTRP_DB_NAME_PO CTRP_DB_USERNAME_PO CTRP_DB_PASSWORD_PO

CTRP_DB_NAME_DW CTRP_DB_USERNAME_DW CTRP_DB_PASSWORD_DW

Environment variables that need to be setup ONLY in local CTRP_DB_HOST_LOCAL CTRP_DB_PORT_LOCAL CTRP_DB_NAME_GRANTS_LOCAL CTRP_DB_NAME_PA_LOCAL CTRP_DB_NAME_PO_LOCAL CTRP_DB_NAME_DW_LOCAL CTRP_DB_USERNAME_GRANTS_LOCAL CTRP_DB_USERNAME_PA_LOCAL CTRP_DB_USERNAME_PO_LOCAL CTRP_DB_USERNAME_DW_LOCAL CTRP_DB_PASSWORD_GRANTS_LOCAL CTRP_DB_PASSWORD_PA_LOCAL CTRP_DB_PASSWORD_PO_LOCAL CTRP_DB_PASSWORD_DW_LOCAL