Multi-armed bandit (MAB) problems, typically modeled as Markov decision processes (MDPs), exemplify the exploration vs. exploitation tradeoff. An area that has motivated theoretical research in MABs is clinical trials, where the application of MAB designs has the potential to improve patient outcomes and reduce drug development costs. For many practical problems of interest, the state space is intractably large, rendering exact approaches to solving MDPs impractical. In particular, settings with delays in observing outcomes that require multiple simultaneous randomizations (as in most practical clinical trials) lead to an expanded state and action space. Approximation methods make it computationally feasible to solve large-scale MDPs. Although some methods are more popular than others, approximation still remains more of an art. In this study, we propose a novel approximation approach that combines the strengths of multiple methods to obtain near-optimal policies for large scale MDPs, with minimal added computational burden. More specifically, our approach uses grid-based state discretization techniques, together with methods to improve approximation accuracy (addition of a risk measure to the objective and bounds on the interpolated value), and simulation-based approaches. Our detailed numerical analysis on relevant datasets shows our proposed design to be almost as good as a fully optimal design and superior to the existing heuristics, given sufficient opportunities to learn. We also demonstrate the value of our design through a retrospective implementation on a practical problem instance, where we find that our design could have reduced the number of failures by 37%.
– (This is joint work with John Birge, University of Chicago) –
Vishal Ahuja is an Assistant Professor of Information Technology and Operations Management (ITOM) at the Southern Methodist University (SMU) Cox School of Business and Assistant Professor (by courtesy), Computer Science and Engineering at SMU. In addition, he holds an appointment as an Adjunct Assistant Professor of clinical sciences at the University of Texas Southwestern Medical Center. Vishal is also a Research Associate, Medical Services at the Veterans Administration Hospital (North Texas Health Care System), and a faculty member at the Center for Global Health Impact at SMU. Vishal joined the Cox School in 2014 after receiving a PhD and an MBA from the Booth School of Business at the University of Chicago. Vishal’s research focuses on developing decision analytic tools that can be implemented easily by healthcare professionals and policymakers to improve patient health, advance the quality of care, and enhance the efficiency of delivery of care. A focus of Vishal’s research is chronic diseases, diabetes in particular, where he intends to implement analytical tools to improve patient outcomes. He teaches MBA courses in operations management and service operations management at the Cox School. To bring relevance to his research, Vishal attempts to draw from his diverse work experience of over 7 years in the corporate sector that includes engineering and managerial roles in the chemical, manufacturing, and consumer goods industry.
For more information about the ESD Seminars Series, please contact Ying Xu at email@example.com.