by Luke Shulman

Open Clinical Standards and Measure Calculation with MITRE

This is part 2 in our series on Open Clinical Standards with MITRE. Here is a link to part 1

Last week, we were fortunate to have been invited up to meet the staff at MITRE’s MITRE CMS Alliance to Modernize Healthcare. If you have never heard of MITRE, they are part of a consortium of Federally-Funded Research & Development Centers. Basically, MITRE serves to explore new technologies and areas for research for the US Government. In healthcare, this has meant MITRE has been at the forefront of created open-source technologies to help the adoption of interoperability standards, health analysis and testing. Many vital technologies underpinning our health IT ecosystem were pioneered at MITRE.

We got a tour of their work that ranges from a new system to categorize cause of death information to a care team “day-at-a-glance” app that provides advanced information about a patients risk markers to facilitate visit planning. In this blog post, I want to cover how MITRE tools can accelerate the work of developers and analysts like myself who want to better understand standards such as FHIR, C-CDA, CQM etc.

If you are a developer starting to build an application that needs to eventually get clinical and quality data, you need to be familiar with these tools which can give you drop-in capability to speed your development.

The Use Case

For this blog post, we will set-up a custom development environment using MITRE’s tools for FHIR and quality measure calculation. We will start by generating realistic clinical EHR Data. We load that into a reference FHIR server and database. Finally, we will run a CMS electronic quality measure (eCQM) across the population. We will do all of this without writing one line of code (MITRE has done it all for us).

To do this we will use five different MITRE packages that all work in harmony with a common FHIR data model. Each on their own would save a developer months of time and money. Just the test-data generator alone means groups can have developer environments with real test data without needing to pay the HIPAA tax.

So, I said no code and thats true. You will need some basic skills with the command line to install the packages and start the servers. If you are into containerization, I have a Vagrant set-up script located on GitHub that will perform this set-up for you.

The MITRE packages we will be using are as follows:

  1. Synthea: The Synthetic Patient Generator. Synthea generates high-quality realistic patients based on statistically valid rules. The patient records are generated in both FHIR and C-CDA formats. You can Synthea in action at SyntheticMass where it was used to realistically create a model of health for the entire state of Massachusetts.
  2. goFHIR Server: MITRE’s generic implementation of the FHIR models and APIs. This is a fully-functional FHIR server backed by MongoDB for data storage and written in GO. Most of MITRE’s projects use this project but wrap the generic services in customization for a specific use case. But the basics of FHIR and over 65 models for FHIR data are implemented here. You don’t need to visit this repo we will be using a customized wrapper for quality measurement.
  3. ECQM Server: This GO server includes all of the FHIR endpoints from the goFHIR server but also adds several custom APIs to support the calculation of quality measures by the cqm-engine. this will be the package you interact with most.
  4. Node CQM Engine: This a framework for ensuring that measures calculate. It is actually a simple polling/queuing server. It polls the mongoDB database for changes in data and recalculates the quality measures accordingly. The quality measure logic is automatically obtained using CMS quality measure tools to download the measure definitions directly from their source.
  5. eCQM Frontend: A web-based application for viewing quality measure results from this stack.

Its a lot of separate parts but the effort is worth it. This architecture emphasizes smaller parts that accomplish small defined tasks. Sure, one massive server/application could do all of these things but that would be much harder to build and it means that the developer or analyst couldn’t adapt that system or its technologies to their own environment. “Do one thing do it well”.

System and Data Prerequisites

The set-up scripts that follow are for a linux based system (Ubuntu 16.04) in my case. This set-up can theoretically be installed on Windows machines as well but I wouldn’t recommend it. Windows users may want to consider a VirtualBox Ubuntu instance.

Base technologies

The various tool sets rely on a range of technologies. So let’s start by installing all of those. The following commands will install Ruby, Nodejs, GoLang, and MongoDB which will support all of the packages.

Let’s start with MongoDB which we will install from the Mongo repository rather than from Ubuntu:

apt-key adv --keyserver hkp:// --recv EA312927
echo "deb xenial/mongodb-org/3.2 multiverse" |  tee /etc/apt/sources.list.d/mongodb-org-3.2.list
sudo apt-get update
sudo apt-get install -y mongodb --allow-unauthenticated

Now you can start the mongodb database:

sudo mongod --fork --logpath /var/log/mongodb.log

Verify that mongo is up and running by opening the Mongo interface and testing it:

show dbs

Now lets install the rest of the languages:

sudo apt-get install -y golang ruby ruby-dev
sudo apt-get install -y nodejs npm redis-server

Some key NPM packages:

npm install -g webpack

I like to use a utility called forever to keep certain scripts up and running “forever” even if I have close the shell.

sudo npm install forever -g

Last step, make sure you have a gopath set as a environment variable this is where go will store all of its code. This should be your something like

export GOPATH=/home/[YOUR USER]/go

ECQM Front-End

This may seem a little funny but I find it easier to actually install the packages in reverse. Building the ecqm front-end, starting the quality measure engine and lastly loading the data. This may seem counter intuitive but it can make things a little bit easier.

So lets clone and checkout the ecqm-frontend repo.

git clone
cd ecqm-frontend

Since this is a node project, we will need a bunch of node packages installed. This will take a minute. Also, Ubunutu uses a different command name for node so the symlink is important.

sudo  ln -s /usr/bin/nodejs /usr/bin/node
npm install

Once complete, we will actually create the frontend assets by running the following webpack command:

NODE_ENV=production webpack --config ./webpack.production.config.js --progress

Assuming this ran correctly check the dist folder: ``bash cd dist ls pwd

After pwd, you should see the full path to this directory that contains the `index.html` file that is the homepage of the application. Keep a note of this path you will need it later. Now return to your parent directory or to your home wherever you keeping all of this code and lets get to work on setting up the calculation engine. 

### Node CQM Engine

Clone and checkout the node-cqm-engine:
git clone
cd node-cqm-engine.git

More node packages

npm install

As I stated above, the genius of this system is that it can process the measures directly from their source format which is the Health Quality Measure Format. Its a bear of complicated XML. Luckily, we get to use the parser that is included the gems we installed above. That will generate a nice JSON representation that can be used calculation.

To tell the engine what measures, you want you have create a “measure bundle” basically a zip file of the HQMF documents that get loaded into MongoDB. Official certified measure bundles are generated by project Cypress for use in the certification and testing regime. You can download the measure bundle directly from You will be prompted for a login. Register for an account at the National Library of Medicine UTS

The following commands will download the measure bundle and also load it into mongoDB to be used for calculation later. Make sure to replace “Homer Simpson’s” credentials with your own:

curl -u hsimpson:DUFFmoesbar! >
nodejs load_bundle.js -b

Verify the measures loaded by opening mongo and querying for the measures:

use FHIR

If the measures loaded, we can start the cqm-engine services. There are two qr_monitor.js which polls the db to see if new calculations are required and calculation_job.js which works a queue to actually do the processing and pulls data from the mongoDB FHIR database.

I start them using forever so that they can run in the background and i can do something else.

forever start calculation_job.js
forever start qr_monitor.js

Wait about 30 seconds and then take a look at the logs to see if they are up:

forever logs calculation_job.js
forever logs qr_monitor.js

ECQM and the FHIR Server

The server that ties all this together is written in go. So before starting these components we want to make sure we are working in the gopath.

mkdir -p $GOPATH/src/
cd $GOPATH/src/
git clone
cd ecqm

Install the prerequisite pacakges. The MITRE project uses “glide” to manage some dependencies along with GOs own management system.

go get
glide install
go install

Now to start the FHIR server, you will need to reference the ecqm-frontend which we built using NodeJS above. Remember the ‘dist’ folder which contains the index.html for the front-end. Yea that one, you will need to tell the server to run and serve that directory. So where I use vagrant my command to run the server is

go run server.go -assets /home/vagrant/ecqm-frontend/dist

On your machine the path will be different. Start the server. The output should list each endpoint available and look something link this:

[GIN-debug] GET    /RecordMatchRunMetrics    --> (6 handlers)
[GIN-debug] GET    /RecordMatchRunLinks/:id  --> (6 handlers)
[GIN-debug] GET    /ptmatch/api/*filepath    -->*RouterGroup).createStaticHandler.func1 (6 handlers)
[GIN-debug] HEAD   /ptmatch/api/*filepath    -->*RouterGroup).createStaticHandler.func1 (6 handlers)
[GIN-debug] GET    /                         -->*RouterGroup).StaticFile.func1 (6 handlers)
[GIN-debug] HEAD   /                         -->*RouterGroup).StaticFile.func1 (6 handlers)
[GIN-debug] GET    /assets/*filepath         -->*RouterGroup).createStaticHandler.func1 (6 handlers)
[GIN-debug] HEAD   /assets/*filepath         -->*RouterGroup).createStaticHandler.func1 (6 handlers)
[GIN-debug] Listening and serving HTTP on :3001

If you scroll up through this output, you will see every single FHIR endpoint such as “Patient”, “Procedure”, “DiagnosticReport” all of these conforming to the FHIR standard.

Using a web-browser visit [http://localhost:3001] and you should see the eCQM engine running as in the picture below:

Loading Data with Synthea

As I said, we are working backward. So now is the time to load some data. Pick a good parent directory and now lets get the synthetic patient generator:

git clone
cd synthea

To match FHIR versions you will need to use the V1.0.0 version

git checkout v1.0.0

Once that is done you can install ruby dependencies using bundle:

sudo gem install bundler
bundle install

Reference the Synthea README for a full list of commands but for now lets generate patients. Their FHIR documents will be placed into output/fhir and their C-CDA documents in output/CCDA

bundle exec rake synthea:generate['./config/Suffolk_County.json']

You should then see all the patients generate year-by-year.

I hope you left your FHIR server running. Assuming you did the next command should be easy.

bundle exec rake synthea:fhirupload[http://localhost:3001]

This will load all of the FHIR documents to the FHIR server we loaded.

You will want to generate a lot of patients so feel free to play with the other generate commands. Now that everything is loaded go back to ECQM and you should see your quality measures running.

ECMQ Measures