newsdev/fec2json

Name: fec2json

Owner: NYT Newsroom Developers

Description: turn fec files into json

Created: 2018-02-13 18:46:30.0

Updated: 2018-05-22 22:29:39.0

Pushed: 2018-04-05 19:43:57.0

Homepage: null

Size: 126

Language: Python

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

fec2json

turn fec files into json

work in progress, not yet trustworthy

The purpose of this library is to turn files from the fec format into json.

Running

Requires python3.x. To install dependencies, pip install -r requirements.txt. We recommend you do that in a virtual enviromnet.

CLI:

fec2json --path path/to/filing

Writes json to stdout.

Takes the following optional commandline arguments

• --fecfile if you have .fec files instead of .csv files (no argument needed, this stores a boolean, we assume it's csv if not specified)
• --filing_id to specify the filing id. If not specified, we assume it is the filename minus the directory path and extension.

Library:

The code can currently be cloned from github and used as a library in another python app. TKTK: pip.

Output

The json will have the following keys:

from the first line of the fec file:

• record_type
• electronic_filing_type
• fec_version_number
• software_name
• software_version
• report_id
• report_type
• header_comment
• amends_filing (pulled from report_id)

from the second line of the fec file

• amendment (whether the file is amended)
• form (the form type minus the N or A)
• remaining fields depend on the form type and come from the relevant fec csv source file

from the itemizations

• itemizations, an object which includes keys for each sked type
• each of those keys points to a list of itemization objects of the relevant type. The keys for those innermost objects are determined by the relevant fec csv source file
• each itemization also includes filing_id of the filing. The filing ID. The filing ID can be provided as a command line argument when running in command line mode. If it is not provided, we assume the filing's path is of the format path/to/filing/filing_id.ext.

For example, a filing might look like this:

ecord_type":"HDR",
lectronic_filing_type":"FEC",
ec_version_number":"8.2",
oftware_name":"NGP",
oftware_version":"8",
eport_id":"",
eport_type":"",
eader_comment":"",
orm_type":"F3N",
iler_committee_id_number":"C00654178",
ommittee_name":"Sara Dady for Congress",
hange_of_address":"",
treet_1":"PO Box 7164",
treet_2":"",
ity":"Rockford",
tate":"IL",
ip":"61126",
eport_code":"YE",
lection_code":"",
lection_date":"",
lection_state":"IL",
lection_district":"16",
tate_of_election":"",
overage_from_date":"20171001",
overage_through_date":"20171231",
reasurer_last_name":"Lashock",
reasurer_first_name":"Gwen",
reasurer_middle_name":"",
reasurer_prefix":"",
reasurer_suffix":"",
ate_signed":"20180131",
ol_a_total_contributions_no_loans":"54485.57",
ol_a_total_contributions_refunds":"255.00",
ol_a_net_contributions":"54230.57",
ol_a_total_operating_expenditures":"72966.66",
ol_a_total_offset_to_operating_expenditures":"0.00",
ol_a_net_operating_expenditures":"72966.66",
ol_a_cash_on_hand_close_of_period":"23164.12",
ol_a_debts_to":"0.00",
ol_a_debts_by":"0.00",
ol_a_individual_contributions_itemized":"30420.00",
ol_a_individual_contributions_unitemized":"22850.57",
ol_a_total_individual_contributions":"53270.57",
ol_a_political_party_contributions":"0.00",
ol_a_pac_contributions":"1100.00",
ol_a_candidate_contributions":"115.00",
ol_a_total_contributions":"54485.57",
ol_a_transfers_from_authorized":"0.00",
ol_a_candidate_loans":"0.00",
ol_a_other_loans":"0.00",
ol_a_total_loans":"0.00",
ol_a_offset_to_operating_expenditures":"0.00",
ol_a_other_receipts":"0.00",
ol_a_total_receipts":"54485.57",
ol_a_operating_expenditures":"72966.66",
ol_a_transfers_to_authorized":"0.00",
ol_a_candidate_loan_repayments":"0.00",
ol_a_other_loan_repayments":"0.00",
ol_a_total_loan_repayments":"0.00",
ol_a_refunds_to_individuals":"255.00",
ol_a_refunds_to_party_committees":"54485.57",
ol_a_refunds_to_other_committees":"96385.78",
ol_a_total_refunds":"255.00",
ol_a_other_disbursements":"0.00",
ol_a_total_disbursements":"73221.66",
ol_a_cash_on_hand_beginning_period":"41900.21",
ol_a_total_disbursements_period":"73221.66",
ol_b_total_contributions_no_loans":"114759.59",
ol_b_total_contributions_refunds":"1055.00",
ol_b_net_contributions":"113704.59",
ol_b_total_operating_expenditures":"88530.47",
ol_b_total_offset_to_operating_expenditures":"0.00",
ol_b_net_operating_expenditures":"88530.47",
ol_b_individual_contributions_itemized":"73548.96",
ol_b_individual_contributions_unitemized":"37360.63",
ol_b_total_individual_contributions":"110909.59",
ol_b_political_party_contributions":"0.00",
ol_b_pac_contributions":"1100.00",
ol_b_candidate_contributions":"2750.00",
ol_b_total_contributions":"114759.59",
ol_b_transfers_from_authorized":"0.00",
ol_b_candidate_loans":"0.00",
ol_b_other_loans":"0.00",
ol_b_total_loans":"0.00",
ol_b_offset_to_operating_expenditures":"0.00",
ol_b_other_receipts":"0.00",
ol_b_total_receipts":"114759.59",
ol_b_operating_expenditures":"88530.47",
ol_b_transfers_to_authorized":"0.00",
ol_b_candidate_loan_repayments":"0.00",
ol_b_other_loan_repayments":"0.00",
ol_b_total_loan_repayments":"0.00",
ol_b_refunds_to_individuals":"1055.00",
ol_b_refunds_to_party_committees":"0.00",
ol_b_refunds_to_other_committees":"0.00",
ol_b_total_refunds":"1055.00",
ol_b_other_disbursements":"2010.00",
ol_b_total_disbursements":"91595.47",
mendment":false,
orm":"F3",
temizations":{
"SchA":[
  {
    "form_type":"SA11AI",
    "filer_committee_id_number":"C00654178",
    "transaction_id":"760417",
    "back_reference_tran_id_number":"",
    "back_reference_sched_name":"",
    "entity_type":"IND",
    "contributor_organization_name":"",
    "contributor_last_name":"Abedrabbo",
    "contributor_first_name":"Kamal",
    "contributor_middle_name":"",
    "contributor_prefix":"",
    "contributor_suffix":"",
    "contributor_street_1":"515 Verona Dr",
    "contributor_street_2":"",
    "contributor_city":"Rockford",
    "contributor_state":"IL",
    "contributor_zip":"611075307",
    "election_code":"P2018",
    "election_other_description":"",
    "contribution_date":"20171228",
    "contribution_amount":"1000.00",
    "contribution_aggregate":"1000.00",
    "contribution_purpose_descrip":"",
    "contributor_employer":"Twins Auto Mall",
    "contributor_occupation":"Principal",
    "donor_committee_fec_id":"",
    "donor_committee_name":"",
    "donor_candidate_fec_id":"",
    "donor_candidate_last_name":"",
    "donor_candidate_first_name":"",
    "donor_candidate_middle_name":"",
    "donor_candidate_prefix":"",
    "donor_candidate_suffix":"",
    "donor_candidate_office":"",
    "donor_candidate_state":"",
    "donor_candidate_district":"",
    "conduit_name":"",
    "conduit_street1":"",
    "conduit_street2":"",
    "conduit_city":"",
    "conduit_state":"",
    "conduit_zip":"",
    "memo_code":"",
    "memo_text_description":"",
    "reference_code":""
  }
],
"SchB":[
  {
    "form_type":"SB17",
    "filer_committee_id_number":"C00654178",
    "transaction_id":"500018114",
    "back_reference_tran_id_number":"",
    "back_reference_sched_name":"",
    "entity_type":"ORG",
    "payee_organization_name":"ActBlue Technical Services",
    "payee_last_name":"",
    "payee_first_name":"",
    "payee_middle_name":"",
    "payee_prefix":"",
    "payee_suffix":"",
    "payee_street_1":"366 Summer St",
    "payee_street_2":"",
    "payee_city":"Somerville",
    "payee_state":"MA",
    "payee_zip":"021443132",
    "election_code":"P2018",
    "election_other_description":"",
    "expenditure_date":"20171107",
    "expenditure_amount":"29.67",
    "semi_annual_refunded_bundled_amt":"0.00",
    "expenditure_purpose_descrip":"Credit Card Processing FEe",
    "category_code":"",
    "beneficiary_committee_fec_id":"",
    "beneficiary_committee_name":"",
    "beneficiary_candidate_fec_id":"",
    "beneficiary_candidate_last_name":"",
    "beneficiary_candidate_first_name":"",
    "beneficiary_candidate_middle_name":"",
    "beneficiary_candidate_prefix":"",
    "beneficiary_candidate_suffix":"",
    "beneficiary_candidate_office":"",
    "beneficiary_candidate_state":"",
    "beneficiary_candidate_district":"",
    "conduit_name":"",
    "conduit_street_1":"",
    "conduit_street_2":"",
    "conduit_city":"",
    "conduit_state":"",
    "conduit_zip":"",
    "memo_code":"",
    "memo_text_description":"",
    "reference_to_si_or_sl_system_code_that_identifies_the_account":""
  }
]