The Open Mainframe Project recently introduced Zorse, a new initiative focused on using AI to improve programming capabilities on mainframe systems, especially in languages like COBOL and JCL. The Zorse project’s goal is to enhance large language models (LLMs) to better understand and generate mainframe-specific code, addressing the current performance gap these models face with legacy programming languages.
The need for Zorse is significant because current LLMs perform comparatively poorly at reading and writing mainframe code because they have not been trained on the appropriate mainframe data. The focus of the Zorse project is on collecting large, production-quality datasets and using them to train LLMs to understand and write mainframe code. An evaluation tool for measuring the performance of LLMs on mainframe programming tasks will also be provided through the Zorse project.
This repository of high-quality, production-grade COBOL and related code, along with an evaluation suite to test LLM performance on tasks specific to mainframe environments will offer mainframe developers a valuable AI tool for interpreting and generating mainframe code.
The Open Mainframe Project
The Open Mainframe Project is a collaborative initiative under the Linux Foundation, created to promote the use, development, and modernization of mainframe technology. Launched in 2015, the Open Mainframe Project seeks to bridge the gap between the mainframe and open-source communities by providing resources, tools, and a support network to make mainframe computing more accessible to modern developers. The project brings together industry stakeholders, including large enterprises, software vendors, and academic institutions, to work on open-source projects that enhance the functionality and interoperability of mainframe.
The first Open Mainframe Project was Zowe, an open source software framework that provides solutions that allow development and operations teams to securely, manage, control, script and develop on the Mainframe like any other cloud platform. Zowe was the first open source project based on z/OS.
There are several other active projects sponsored by the Open Mainframe Project including GenevaERS (a single-pass optimization engine for data extraction and transformation on z/OS ) and education initiatives including the COBOL Programming Course.
Zorse is on of the newer Open Mainframe Project projects, currently in the incubation phase, which means it is in the process of establishing governance, diversity and sustainability within their communities. Nevertheless, the focus of Zorse on improving LLMs for mainframe tasks aligns with the Open Mainframe Project’s broader goals of modernizing mainframe development and expanding the accessibility of legacy systems to new generations of developers and AI technologies.
Can AI Replace Developers?
Although AI cannot yet perform all of the development tasks of a seasoned professional programmer, LLMs can generate code that approaches production quality for specific, well-defined tasks. However, their current limitations mean they aren’t yet capable of consistently generating complex, fully production-ready programs without human intervention.
The quality of LLM-generated code depends on several factors, including the clarity of the instructions, the availability of similar examples in the training data, and the complexity of the task. For example, LLMs can effectively produce boilerplate code, data processing scripts, or simple web applications, but they struggle with intricate requirements, optimization, security, and performance standards typically required in production-grade software.
Key limitations of LLMs in generating production-quality code include:
- Context and Requirement Understanding: Context and Requirement Understanding: LLMs lack the capability to fully understand intricate business requirements, often essential for production code. They interpret input as text patterns rather than as deeply contextual requirements, which can lead to oversimplifications or omissions of critical functionality.
- Error Handling and Debugging: LLMs frequently generate code with minor syntax issues or logical errors that need debugging. Although they’re improving, LLMs are not yet reliable at ensuring error-free code in complex applications, particularly when it comes to edge cases and exception handling.
- Security and Compliance: Production-quality code must meet security and compliance standards, particularly in enterprise or government settings. LLMs aren’t inherently aware of these standards, and their code can contain vulnerabilities (such as SQL injection risks) unless carefully guided by developers.
- Scalability and Optimization: LLMs typically do not account for efficiency optimizations or scalability considerations unless explicitly instructed. They are generally better suited for generating functionally correct code rather than highly optimized code that can handle large-scale production loads.
These limitations exist regardless of the platform, but are exacerbated on the mainframe due to the lack of models trained on mainframe data. Projects like Zorse can help narrow these gaps for mainframe applications by fine-tuning LLMs on high-quality, domain-specific datasets, allowing models to better interpret and generate mainframe-specific code. However, even with improvements, LLM-generated code still requires review and testing by skilled developers to reach production standards reliably.
LLMs are valuable for accelerating code creation, reducing repetitive tasks, and supporting developers, but they are best used in a collaborative setting rather than as a standalone tool for end-to-end production code generation.
The Bottom Line
Zorse has the potential to have a transformative impact on mainframe coding and development using AI. By addressing critical challenges LLMs face when working with mainframe languages like COBOL, JCL, and others, Zorse can help to overcome the current limitations of GenAI for mainframe programming. One of Zorse’s primary contributions will be the creation of a high-quality dataset of production-level COBOL code, enabling LLMs to better interpret, generate, and maintain mainframe applications. This dataset is expected to provide the context and examples that LLMs currently lack, given the scarcity of mainframe-specific code available in typical training sets.
Furthermore, Zorse’s evaluation suite should make it simpler to measure progress in the application of AI to mainframe programming, helping developers assess which models are most effective at mainframe coding tasks. By enabling LLMs to better understand COBOL syntax, semantics, and unique programming conventions, Zorse should bolster the reliability of AI as a trustworthy tool for mainframe developers, ultimately enhancing productivity and easing mainframe application modernization efforts.
Ultimately, Zorse should help to enable organizations as they build Artificial Intelligence coding tools that will boost the productivity of mainframe software engineers.
