Sounds interesting? Here's the specific research question:
Benchmarking of LLM features related to OR software to develop an OPTANO decision catalog for various use cases
- OPTANO already solves the actual operations research scenario. The research question, therefore, is not whether LLMs can replace the optimization problem, but rather in which related use cases they provide reliable added value.
- The goal is to examine how well various LLM features perform in typical use cases such as data science, what-if analyses, explainability, interpretation of results, and user interaction.
- It also remains to be seen which technical „approaches“ are appropriate depending on customer requirements—for example, simple prompting versus retrieval, single-agent versus multi-agent—and how these variations affect cost, runtime, and quality.
Here is how we envision this methodology:
- Development of a benchmark framework for several clearly defined use cases related to an existing OPTANO scenario, such as data analysis, scenario comparison, explanation of solutions, or answering technical inquiries.
- A systematic comparison of various LLM setups based on defined metrics such as response quality, factual accuracy, processing time, cost, robustness, and the engineering effort required.
- Development of a practical decision-making framework with reusable base templates from which appropriate LLM architectures can be selected based on the use case and customer requirements.
You'll need the following knowledge:
- Strong knowledge of mathematical optimization or operations research
- Programming experience in Python and an interest in experimental evaluation and benchmark design
- Basic Knowledge of LLMs, Prompting, Retrieval, and Agent-Based Architectures