Medicare data ingestion and processing pipeline

Source code

Workflow

Description

This workflow processes raw Medicare data. We assume that the data for each year is in a separate set of SAS DAT files accompanied by FTS. For each year we expect at least two tables: patient summary and inpatient admissions.

NB: Input files must be organized within the dicrectory given in the input parameter in a certain way. Immediate parent folder for each file should be named as the year of the data it contains. Example:

data/
  a/
    b/
      2011/
      2013/
    d/
      2017/

See Medicare Files Handling for data processing details.

Inputs

Name	Type	Default	Description
database	File		Path to database connection file, usually database.ini
connection_name	string		The name of the section in the database.ini file
input	Directory		A path to directory, containing folders with unpacked CMS files. The tool will recursively look for data files according to provided pattern. Immediate parent folder for each file should be named as the year of the data it contains, e.g. a/b/c/2017/mbsf_abcd_xyzacdfrtwe_request12345.fts

Outputs

Name	Type	Description
initdb_log	File
initdb_err	File
load_raw_log	File
load_raw_err	File
registry	File
d_create_log	File
d_create_err	File
d_index_log	File
d_index_err	File
d_vacuum_log	File
d_vacuum_err	File
ps_create_log	File
ps_create_err	File
ps2_create_log	File
ps2_create_err	File
bene_view_log	File
bene_view_err	File
bene_table_create_log	File
bene_table_index_log	File
bene_table_vacuum_log	File
bene_table_create_err	File
bene_table_index_err	File
bene_table_vacuum_err	File
enrlm_view_log	File
enrlm_view_err	File
enrlm_table_create_log	File
enrlm_table_index_log	File
enrlm_table_vacuum_log	File
enrlm_table_create_err	File
enrlm_table_index_err	File
enrlm_table_vacuum_err	File
ip_create_log	File
ip_create_err	File
adm_create_log	File
adm_create_err	File
adm_populate_log	File
adm_populate_err	File
adm_index_log	File
adm_index_err	File
adm_vacuum_log	File
adm_vacuum_err	File
qc_ev_create_log	File
qc_ev_create_err	File
qc_av_create_log	File
qc_av_create_err	File
qc_enrollmen343_create_log	File
qc_enrollmen343_index_log	File
qc_enrollmen343_vacuum_log	File
qc_enrollmen343_create_err	File
qc_enrollmen343_index_err	File
qc_enrollmen343_vacuum_err	File
qc_admission697_create_log	File
qc_admission697_index_log	File
qc_admission697_vacuum_log	File
qc_admission697_create_err	File
qc_admission697_index_err	File
qc_admission697_vacuum_err	File
grant_log	File
grant_err	File

Steps

Name	Runs	Description
initdb	initdb.cwl	Ensure that database utilities are at their latest version
load_raw_data	load_raw_medicare.cwl	Load raw CMS Medicare data into the database
enrollments	medicare_beneficiaries.cwl	Process beneficiaries enrollment data
admissions	medicare_admissions.cwl	Process medicare inpatient admissions (aka Medpar) data
qc	medicare_qc.cwl	Build QC Tables
grant	alter_database.cwl	Grants read access to the members of NSAPH group for newly created or updated tables