Triage

An open source machine learning toolkit to help data scientists, machine learning developers, and analysts quickly prototype, build and evaluate end-to-end predictive risk modeling systems for public policy and social good problems.

Triage lets you focus on the problem you’re solving and guides you through design choices you need to make at each step of the machine learning pipeline.

Why we created Triage

We created Triage in response to commonly occurring challenges in the development of machine learning systems for public policy and social good problems. While many tools (sklearn, keras, pytorch, etc.) exist to build ML models, an end-to-end project requires a lot more than just building models.

Building systems with predictive models that are going to be used in production require making many design decisions that need to match with how the system is going to be used. These choices then get turned into modeling choices and code.

We need to answer questions such as:

  • What should be included in the data (cohort selection)?
  • What should our rows consist of (unit of analysis)?
  • What is our label? What outcome are we predicting or estimating and over what period of time?
  • How should we generate and incorporate time and space varying features (spatiotemporal explanatory variables)?
  • How do we deal with time in our training, selection, and validation process?
  • Which models/methods and associated hyper-parameters should we try?
  • What evaluation metrics do we use to compare and select models?
  • How do we compare and evaluate models over time?
  • How should we interpret and explain the models and it’s predictions to the user in the loop?
  • How do we ensure fairness and equity in our system?
  • How to we understand and communicate intervention lists generated by these models?

These questions are critical but complicated and hard to answer apriori. Even when these design choices are made, we still have turn these choices into code throughout the course of a project. Triage is designed around these questions and generates a set of data matrices, models, predictions, evaluations, and analysis that makes it easier for data scientists to select the best models to use.

Triage aims to help solve these problems by:

  1. Guiding users (data scientists, analysts, researchers) through these design choices by highlighting operational use questions that are important.
  2. Providing interfaces to these different phases of a project, such as an Experiment. Each phase is defined by a configuration (corresponding to a design choice) specific to the needs of the project, and an arrangement of core data science components that work together to produce the output of that phase.

Each of these components require careful design choices to be made. Triage facilitates this decision-making process for programmers and developers, with a special focus on tackling data science, AI, and ML problems in public policy and social impact.

What components are inside Triage?

Timechop

Generate temporal validation time windows for matrix creation

When is it helpful?

In predictive analytics, temporal validation can be complicated. Timechop takes in high-level time configuration (e.g. lists of train label spans, test data frequencies) and returns all matrix time definitions.

Collate

Aggregation SQL Query Builder

When is it helpful?

Collate allows you to vary both the spatial and temporal windows of a query to easily specify and execute statements like “find the number of restaurants in a given zip code that have had food safety violations within the past year.”

Architect

Plan, design and build train and test matrices

When is this useful?

The Architect helps you properly organize data into design matrices in order to run classification algorithms

Catwalk

Training, testing, and evaluating machine learning classifier models

When is it helpful?

Catwalk allows you to train classifiers on large set of design matrices, test and temporally cross-validate them, generating evaluation metrics about them.

Results Schema

Generate a database schema suitable for storing the results of modeling runs

When is it helpful?

The results schema stores the results of modeling runs in a relational database with different options for table creation and for passing database credentials

Audition

When is it helpful?

Audition introduces a structured, semi-automated way of filtering models based on what you consider important. It formalizes this idea through selection rules that take in the data up to a given point in time, apply some rule to choose a model group, and then evaluate the performance (regret) of the chosen model group in the subsequent time window.

Bias and Fairness Audit

Audit models for fairness and bias to explore possible tradeoffs with efficiency and/or effectiveness

When is it helpful?

Incorporates Aequitas, an open source bias audit toolkit for machine learning developers, analysts, and  policymakers to audit machine learning models for discrimination and bias, and make informed and equitable decisions around developing and deploying predictive risk-assessment tools.

Post-Modeling

Allows the user to dig deeper and explore the models selected by audition

When is it helpful?

Gives the user a better understanding of and comparison across the models selected by Audition. Produces comparisons using feature importances and the predictions of the models.

What do you need to get started with Triage?

Case Studies

More details on the github repository

Reducing Early School Dropout Rates in El Salvador

Triage was used to build an Early Warning System to identify students at risk of not coming back to school in collaboration with the Ministry of Education in El Salvador. These predictions were then used to understand what policy interventions to allocate to which schools to increase retention.

More details on the github repository
More details on the github repository

Improving outcomes for people living with HIV by increasing retention in care

Triage was used to build an Early Warning System to identify people living with HIV who may not be retained in care and prioritizing them for retention interventions. This work was done in two different contexts: 1) with the University of Chicago HIV Clinic at the time of a patient's appointment, and 2) with the Chicago Department of Public Health to inform the proactive retention activities.

More details on the github repository
More details on the github repository

Improving Workplace Safety through Proactive Inspections in Chile

Triage was used to build an Inspection prioritization tool in collaboration with Dirección del Trabajo, Chile’s labor department, to proactively inspect workplaces to reduce the risks workers face.

chile

More details on the github repository

The Team

Triage was initially created by the Center for Data Science and Public Policy at the University of Chicago and is now moved to Carnegie Mellon University. Our goal is to further the use of AI, ML, and data science in policy research and practice. Our work includes educating current and future policymakers, doing data science projects with government, nonprofit, academic, and foundation partners, and developing new methods and open-source tools that support and extend the use of data science for public policy and social impact in a measurable, fair, and equitable manner.

To contact the team, please email us at rayid @ cmu dot edu

Interested in building and deploying open source, data-driven machine learning systems for governments and social good that are effective and result in equitable outcomes?

Talk to us. We’re building a series of  tools, trainings, and methodologies for governments and non-profits across health, criminal justice, education, social services, economic development, workforce development, and public safety.