Our previous post, MCP on the Mainframe Already Runs, covered the architecture and governance of SteelFrame's MCP server: the tools, the scopes and the audit trail. This post is the practical companion. Instead of driving SteelFrame with our own harness, we handed the MCP server to an AI client we did not build and asked it to find out, in its own way, whether SteelFrame behaves like a mainframe.
The client was Grok Bot, the Grok desktop app from xAI. We asked it for a quick test of SteelFrame's behavioural conformance with a real mainframe in six areas: COBOL compile/link/run, JCL/JES2 job execution and spool, CICS-style transactions, VSAM organizations, EBCDIC and COMP-3. Grok chose the tests, wrote the programs and JCL, called the tools and graded the results. We only read the reports. The screenshots below are Grok's own summaries from a session on 18 September 2026, cropped.
SteelFrame is our mainframe-compatible environment: an independent clean-room implementation, not IBM z/OS, and not affiliated with or endorsed by IBM. It runs COBOL, JCL through a JES2-class spool, CICS-compatible transactions, VSAM and a DB2-compatible SQL engine, built to the published semantics.
Why an outside client matters
When we test SteelFrame, we know where the edges are, and we pick tests without meaning to. A model from another lab does not. Its only view of SteelFrame is the MCP tool catalog (names, descriptions, input schemas), and its idea of "correct" comes from what it learned about IBM mainframes, not from us. We built nothing for Grok. If it can still discover the tools, write a sensible test plan and get plausible mainframe answers back, then the MCP surface works for any client, and the behaviour matches what an informed outsider expects.
Setup: how Grok connected
Nothing in the setup was special to Grok. The
SteelFrame MCP
documentation describes two doors onto the same engine: a local stdio bridge
(bin/steelframe-mcp.mjs, zero dependencies, Node 18+) that fetches its tool catalog
from the engine at startup, and a remote endpoint, POST /mcp on the hosted engine
(JSON-RPC, bearer authentication), for remote-native clients. Either way the client authenticates
with a scoped API token or a userid logon, and scope checks and audit are the same as on the REST
API because both doors dispatch through the same call path.
Grok worked on the persistent lane, where results land in the shared store and show
up in the SteelFrame web interface. It ran as the hosted estate's demonstration userid,
IBMUSER, which appears in its reports, and its jobs carry the (MCP)
accounting field in the spool. No credentials appear in this post or the screenshots.
The results at a glance
Grok ran one warm-up and five suites. Across the suites it reported 45 passes, 1 fail and 1 skip out of 47 checks. The table shows what it tested in each area and what "conformance" means for that area: the mainframe behaviour a check is actually looking for.
| Area | What Grok tested | Grok's result | What conformance means here |
|---|---|---|---|
| Warm-up | Compile, link and run a HELLO COBOL program | Both jobs max CC 0; SYSOUT HELLO FROM STEELFRAME MCP |
The basic edit, compile, link and run loop works end to end through the spool |
| COBOL compile / link / run | Compile-check, good and bad compiles, run, a deliberate abend | 5 of 5 PASS | Diagnostics with line numbers and IBM-style message IDs; RCs flow into max CC; abends surface as system completion codes |
| JCL / JES2 / spool | Submit and wait, multi-step jobs, hold and release, JES files, step SYSOUT, purge | 8 of 8 PASS | Jobs move through JES2 states (INPUT-HOLD, OUTPUT); purged jobs are gone |
| CICS-style transactions | CSD, terminal sessions, AID keys, CardDemo sign-on, scripted runs, cics_link |
8 PASS, 1 FAIL (the CardDemo sign-on; cause found and fixed, see below) | Terminal, AID-key and task semantics; LINK round-trips a COMMAREA |
| VSAM organizations | KSDS, ESDS, RRDS, a CardDemo KSDS (read-only), delete, negative cases | 6 PASS, 0 FAIL, 1 SKIP (RRDS data load) | Key order, duplicate-key reporting, entry-sequenced reads, IDCAMS-style errors |
| EBCDIC | CCSID 37 bytes, round trips, data at rest, COBOL READ/DISPLAY, code-page sensitivity | 6 of 6 PASS | Documented EBCDIC bytes, stored as EBCDIC; code pages differ where they should |
| COMP-3 (packed decimal) | Sign nibbles, lengths, scales, round trips, COBOL arithmetic, ROUNDED, SIZE ERROR, stress set | 12 of 12 PASS (stress 24 of 24 exact) | Sign nibbles C/D/F; ceil((digits+1)/2) bytes; exact decimals; COBOL rounding rules |
What Grok actually did, area by area
The warm-up: HELLO, compiled and run
Grok started small. It compiled a HELLO program (COBOL step RC 0, link-edit RC 0, load
module in IBMUSER.LOADLIB(HELLO)) and ran it in a second job, step GO RC 0. Its
verdict: "smoke test passed".
Grok's first report: compile and link, then a separate run job, both max CC 0.
COBOL, JCL, JES2 and spool: 13 of 13
Next came one suite for the whole batch path, provoking the bad paths as well as the good
ones. check_source passed clean source with rc=0 and
rejected a program with an undefined field at rc=8, naming line 4. A full compile of broken
source ended at max CC 12 with IGYPS2121-S and IGYDS0011-S
diagnostics. A program that divides by zero ended in S0CB ("surfaced honestly", in Grok's words). On the JES2 side it submitted a job with TYPRUN=HOLD, saw it
wait in INPUT-HOLD, released it, and watched it reach OUTPUT. It read JESMSGLG
(system ZOSW, node STEELFRM), read step SYSOUT by its
STEP.DD handle, and confirmed that a purged job returned 404 afterwards.
The COBOL half of the batch suite, including the two deliberate failures.
JES2 and spool checks, and Grok's practical note on parallel compiles.
Grok added one practical note: parallel compiles against the same shared source library can be refused with "dataset in use", so it advised running them one at a time or giving each run its own high-level qualifier. We spot-checked three of the job numbers it quoted (the HELLO run, the failed compile and the S0CB abend), and their step return codes in the spool match its report.
CICS-style transactions: 8 of 9
For CICS, Grok used the terminal tools: open a headless session, send one AID key at a time,
read screens back as named BMS fields. It confirmed the CSD maps CC00 to
COSGN00C and defines the CardDemo mapsets; that sessions open and close cleanly; that
CLEAR and ENTER on a blank terminal leave no hung task; and that a missed expect is
reported as data (expectMet=false) with the session still usable. A scripted run
returned the whole screen transcript in one call, cics_link through the CPIH mirror
returned the COMMAREA with no abend, and a keyboard locked by a running task unlocked when the
task ended.
The one fail was the CardDemo sign-on. In Grok's run, CC00 ended with an APCT
abend in COSGN00C. APCT is the code CICS gives when a program cannot be loaded. The
cause was on our side, not in the CICS behaviour: the CardDemo load modules on the hosted estate
had been link-edited for a different host platform, and the region correctly refused to load them,
as real CICS would. We relinked them for the host on 22 September, and CC00 now signs
on and reaches the CardDemo main menu.
The CICS suite as Grok reported it, fail included.
VSAM organizations: 6 passed, 1 skipped
Grok defined a test KSDS (key length 8, record length 80), loaded three keys, read them back
in key order, and checked that a duplicate key is reported and an upsert updates in place. An ESDS
behaved as entry-sequenced, with no key. The CardDemo account file
(AWS.M2.CARDDEMO.ACCTDATA.VSAM.KSDS: 50 records, key length 11) passed read-only
checks. Deleted clusters were uncatalogued, and a duplicate name or bad key length returned
IDC-style errors. Its summary: KSDS and ESDS "look production-usable".
RRDS was the skip. The define and the catalog information worked. The data round trip did not
happen, because the two MCP loading tools Grok tried (load_dataset and
write_records) did not accept its RRDS input in that form. It suggested IDCAMS REPRO
as the route for a follow-up run.
VSAM: KSDS and ESDS exercised, RRDS defined but not loaded, CardDemo left untouched.
EBCDIC: 6 of 6
This is where a compatible environment is most likely to be subtly wrong, and Grok checked
it at the byte level. HELLO encoded to C8C5D3D3D6 and the digits to
F0–F9, which are the CCSID 37 values. Text written with
write_dataset was stored as EBCDIC, not ASCII. A COBOL program it compiled read that
EBCDIC file and displayed the expected text. For code-page sensitivity it used the characters
£€¬, which is a well-chosen probe: CCSID 37, 1140 (37 with the euro) and 285 (UK)
gave different bytes as expected, and characters with no mapping became the SUB byte
X'3F'. Its verdict: "looks solid".
EBCDIC at the byte level, including the code-page probe.
COMP-3: 12 of 12, and 24 of 24 exact
Grok went deepest on packed decimal. It checked the sign nibbles (+ is C,
- is D, unsigned PIC 9(n) COMP-3 is F). It checked exact round trips of
values such as -0.01 and 99999.99 through both a sequential file and a
KSDS. It checked the byte-length rule, ceil((digits+1)/2), for every size from S9(1) to S9(15),
and scaled fields. Then it made COBOL compute on packed fields: all four operations, 10/6
giving 1.66 truncated and 1.67 with ROUNDED, and an ON SIZE ERROR branch that fired
when it should. A mixed record (EBCDIC key, packed middle, flag byte) read correctly from COBOL, a
24-value stress set came back exact, and CardDemo's own export data decoded without error. Its
verdict: "conformant".
Packed decimal, from sign nibbles to COBOL ROUNDED and SIZE ERROR.
What stood out
- It tested failure, not just success: an undefined field, a compile that had to fail,
a divide-by-zero abend, a CICS
expectmeant to miss, duplicate VSAM names. Conformance lives in the unhappy paths. - It checked bytes, not just output.
C8C5D3D3D6,…7D,X'3F': these are the checks that catch an environment that looks fine on screen and is wrong on disk. - It cleaned up after itself and listed the few members it left in the source and load libraries.
What this test is, and what it isn't
Your licensed system remains the sole conformance authority for your estate. For a real migration the right tool is SteelFrame's reference oracle workflow: it returns joblogs, decoded records beside their raw bytes, screens field by field and state dumps with checksums, and your harness does the comparing. The architecture and governance behind that design are in MCP on the Mainframe Already Runs.
Try it yourself
You can repeat this with your own MCP client. Setup, authentication, scopes and a live tool reference are public:
- SteelFrame (mainframe-compatible): steelframe.cobolstack.com/mcp-docs.html ↗
- SteelFrame X (IBM i-compatible, AS/400-style: CL, RPG, COBOL, DDS, DB2 for i, 5250 sessions): steelframex.cobolstack.com/mcp-docs.html ↗
Access is issued per organisation, with scopes, after a conversation. Ask for evaluation access, bring the conformance question you would most like an agent to answer, and ask it for byte-level evidence, as Grok did.
Results are Grok's own reports from a single session on 18 September 2026, shown in cropped screenshots. We spot-checked three of the job numbers it quoted against the hosted estate's spool, and they matched. Grok's pass/fail judgements are its own and are not a formal certification. IBM, z/OS, CICS, Db2, VSAM, IBM i, AS/400, RPG, RACF, Grok, xAI and other marks are trademarks of their respective owners. CobolStack is not affiliated with, endorsed by, or sponsored by IBM or xAI. SteelFrame and SteelFrame X are independent clean-room mainframe- and IBM i-compatible implementations that conform to the published specifications but are not identical to them; your licensed system remains the sole conformance authority.