Chris Simpkins - Research

Research

My Dissertation (Due to be completed in April 2013)

Integrating Reinforcement Learning into a Programming Language

My Thesis: Improving the compositionality of modules in modular reinforcement learning (MRL) and integrating MRL into a programming language supports reuse in adaptive agent software engineering. This thesis claims that (1) MRL can be extended to support compositionality in modular reinforcement learning agents by uncoupling the reward scales of the modules that comprise the agents, and (2) integrating modular reinforcement learning into a programming language supports adaptive agent programming by reducing the effort required to adapt agent programs to new domains, enabling new forms of software reuse.

Compositionality, an essential property of modularity in software engineering, allows components to be reused in the construction of new systems. In the case of a modular reinforcement learning agent compositionality means being able to reuse behavior modules in new agents without modifying the modules. The current state of the art in modular reinforcement learning supports decomposition but not composition, or module reuse. This dissertation contributes a reformulation of MRL that supports composition by uncoupling the reward scales of the modules that comprise a modular reinforcement learning agent, and an algorithm that solves it.

An excellent way to support software engineering is with practical, usable programming languages. The second major contribution of this dissertation is an embedded domain-specific language and framework in Scala that integrates MRL. We show how this integration is useful to software engineers writing practical adaptive agent software by applying AFABL to non-player character (NPC) programming in games and agent simulations and measuring the benefit of integrated MRL compared to traditional programming. This application to practical software engineering problems distinguishes AFABL from previous work in integrating RL into programming languages such as ALisp.

Past Projects

Mutual Information-Maximizing Input Clustering (MIMIC) for Antenna Design

Seth Markle, Denis Bueno, and I applied Charles Isbell's MIMIC algorithm to the design of planar array antennas. This work compared MIMIC to a previously published and very successful genetic algorithm approach employed by GTRI's Signature Technology Lab. We found MIMIC to be far superior in terms of execution time due to the computationally expensive evaluation function of real-world antennas. Software and publication to be published soon.

Deriving Scientometric Information by Text Mining Scientific Publications

This work was in service of the research of Dr. Chiara Franzoni, an economist specializing in scientometrics. I applied text mining methods to a database of publication abstracts and biographical information for members of the American Physical Society. Our initial publication focused on finding patterns of specialization (in topic or research method) over time.

Using Optimization Algorithms to Design Neural Networks

This work was part of a larger project led by Dr. Paul Kemper in GTRI's Signature Technology Lab. The project sought effective machine learning algorithms for identifying RF emitters. In an effort to build on previously successful neural network approaches, Denis Bueno and I applied genetic and MIMIC optimization techniques to the design of neural network architectures.

MAT (MAT Analysis Tool): A GUI Analysis Platform for Materials Characterization Experiments

The MAT software package is a modular GUI platform for storing and alayzing materials experiments. I designed a data management system, a flexible user interface for displaying multiple experiments, and a modular plug-in architecture that enables the straightforward addition of new analysis modules, thereby leveraging the data storage and allowing analysts to easily compare experiment data in multiple ways in a single desktop application.

SACRE BLEU: Software-Assisted Content Review Based on Lanuage (English) Understanding

The SACRE BLEU project sought to develop software that identified information of interest in arbitrary human-language text, where information of interest is determined by both policy and precedent. The system is intended for application to the problem of content review for classified information where release officers must aprove the sharing of vast quantities of information on often very short deadlines. My role was to investigate user-adaptive methods and help manage the project, in particular the overall architecture and evaluation of technology components.

Casebook: An Adaptive Problem-Based Learning Environment

Casebook is Dr. Ashwin Ram's web-based platform for applying Problem-Based Learning in the classroom. By providing a suite of collaboration and research tools, and guiding students through the process of PBL, Casebook solves the primary obstacle to adoption of PBL in real-world classroms: the scalability of limited teacher-facilitator resources. Traditional PBL requires a great deal of involvement from the teacher to facilitate the process. Casebook takes care of a great deal of the details so that PBL can be effectively applied in real-world classrooms. I implemented software engineering practices, deployed and maintained the system for classroom research studies, and helped design Casebook 2, which is currently in development.

Mutual Information-Maximizing Input Clustering (MIMIC) for Antenna Design
	Seth Markle, Denis Bueno, and I applied Charles Isbell's MIMIC algorithm to the design of planar array antennas. This work compared MIMIC to a previously published and very successful genetic algorithm approach employed by GTRI's Signature Technology Lab. We found MIMIC to be far superior in terms of execution time due to the computationally expensive evaluation function of real-world antennas. Software and publication to be published soon.
Deriving Scientometric Information by Text Mining Scientific Publications
	This work was in service of the research of Dr. Chiara Franzoni, an economist specializing in scientometrics. I applied text mining methods to a database of publication abstracts and biographical information for members of the American Physical Society. Our initial publication focused on finding patterns of specialization (in topic or research method) over time.
Using Optimization Algorithms to Design Neural Networks
	This work was part of a larger project led by Dr. Paul Kemper in GTRI's Signature Technology Lab. The project sought effective machine learning algorithms for identifying RF emitters. In an effort to build on previously successful neural network approaches, Denis Bueno and I applied genetic and MIMIC optimization techniques to the design of neural network architectures.
MAT (MAT Analysis Tool): A GUI Analysis Platform for Materials Characterization Experiments
	The MAT software package is a modular GUI platform for storing and alayzing materials experiments. I designed a data management system, a flexible user interface for displaying multiple experiments, and a modular plug-in architecture that enables the straightforward addition of new analysis modules, thereby leveraging the data storage and allowing analysts to easily compare experiment data in multiple ways in a single desktop application.
SACRE BLEU: Software-Assisted Content Review Based on Lanuage (English) Understanding
	The SACRE BLEU project sought to develop software that identified information of interest in arbitrary human-language text, where information of interest is determined by both policy and precedent. The system is intended for application to the problem of content review for classified information where release officers must aprove the sharing of vast quantities of information on often very short deadlines. My role was to investigate user-adaptive methods and help manage the project, in particular the overall architecture and evaluation of technology components.
Casebook: An Adaptive Problem-Based Learning Environment
	Casebook is Dr. Ashwin Ram's web-based platform for applying Problem-Based Learning in the classroom. By providing a suite of collaboration and research tools, and guiding students through the process of PBL, Casebook solves the primary obstacle to adoption of PBL in real-world classroms: the scalability of limited teacher-facilitator resources. Traditional PBL requires a great deal of involvement from the teacher to facilitate the process. Casebook takes care of a great deal of the details so that PBL can be effectively applied in real-world classrooms. I implemented software engineering practices, deployed and maintained the system for classroom research studies, and helped design Casebook 2, which is currently in development.