Reinforcement learning environments where an agent is dropped onto a running multi-service system with something broken or missing, and has to make the platform genuinely work. Grading restarts everything from the agent's own configuration, so a hand-started process or a hand-edited value scores nothing.
Pass rates are measured on the shipped environments rather than estimated. Across every model evaluated, 1 environment run reached a perfect score. That headroom is the point of the set.
Each environment is graded twice from source: once on the untouched broken host, and once with the reference solution applied. A stub that scores above zero would mean the task was already partly done; a reference that scores below one would mean the task is not fully solvable.
Read this before quoting the gate table. The supplier's own soundness document shows a sample of environments all scoring 0.0000 on the stub and 1.0000 on the reference. Across the full set that holds for the stub, but 18 of 89 environments ship no gate evidence at all, and 1 reference run scores below 1.0 (ci_pipeline_reconstruct at 0.9630).
| Environment | Type | Difficulty | GPT-5.5 | GPT-5.6 Sol | Opus 5 | Stub | Ref |
|---|---|---|---|---|---|---|---|
| ci_pipeline_reconstruct devops iac · sento-devops | pipeline reconstruction | hard | 47% k=3 | – | – | 0.00 | 0.96 |
| miso_stack_repair devops iac · miso-devops | deployment repair | hard | 67% k=3 | 67% k=3 | – | 0.00 | 1.00 |
| observability_reconstruct_blind devops iac · sento-devops | infrastructure reconstruction | hard | – | 0% k=8 | 100% k=3 | 0.00 | 1.00 |
| openelis_stack_repair devops iac · openelis-devops | deployment repair | hard | 33% k=3 | 50% k=4 | – | 0.00 | 1.00 |
| tf_aws_drift devops iac · aws-devops | iac drift reconciliation | hard | 56% k=8 | – | – | 0.00 | 1.00 |
| tf_aws_migrate devops iac · aws-devops | iac migration data safety | hard | 33% k=8 | – | – | 0.00 | 1.00 |
| tf_az_stale_runbook devops iac · azure-devops | iac reconstruction stale docs | hard | 50% k=8 | – | – | 0.00 | 1.00 |
| tf_infra_core_repair devops iac · sento-devops | iac reconstruction | hard | 20% k=3 | – | – | 0.00 | 1.00 |
| topology_reconstruct_blind devops iac · sento-devops | infrastructure reconstruction | hard | 33% k=3 | – | – | 0.00 | 1.00 |
| topology_reconstruct_flow_stack devops iac · sento-devops | infrastructure reconstruction | hard | – | 33% k=3 | – | 0.00 | 1.00 |
| aria_acp8_codec feature reconstruction · aria | vector reconstruction f2p | hard | – | – | – | 0.00 | 1.00 |
| aria_acp8_codec__spec feature reconstruction · aria | vector reconstruction f2p | hard | – | – | – | 0.00 | 1.00 |
| compute_engine_symbolic feature reconstruction · compute-engine | repo unit test f2p | hard | 22% k=6 | – | – | 0.00 | 1.00 |
| compute_engine_symbolic__spec feature reconstruction · compute-engine | repo unit test f2p | hard | 8% k=6 | – | – | 0.00 | 1.00 |
| dominion_engine feature reconstruction · ruby-dominion | feature implementation f2p | hard | 17% k=4 | – | – | 0.00 | 1.00 |
| dominion_engine__spec feature reconstruction · ruby-dominion | feature implementation f2p | hard | 17% k=4 | – | – | 0.00 | 1.00 |
| fineract_progressive_emi_engine feature reconstruction · fineract | feature implementation f2p | hard | 67% k=6 | – | – | 0.00 | 1.00 |
| fineract_progressive_emi_engine__spec feature reconstruction · fineract | feature implementation f2p | hard | 59% k=10 | – | – | 0.00 | 1.00 |
| flow_notifications_subsystem feature reconstruction · sento-flow-backend | feature implementation f2p | hard | 39% k=5 | – | – | 0.00 | 1.00 |
| flow_notifications_subsystem__spec feature reconstruction · sento-flow-backend | feature implementation f2p | hard | 46% k=5 | – | – | 0.00 | 1.00 |
| flow_workflows_activities_subsystem feature reconstruction · sento-flow-backend | feature implementation f2p | hard | 31% k=10 | – | – | 0.00 | 1.00 |
| flow_workflows_activities_subsystem__spec feature reconstruction · sento-flow-backend | feature implementation f2p | hard | 35% k=5 | – | – | 0.00 | 1.00 |
| goby_battle_engine feature reconstruction · goby | feature implementation f2p | hard | – | – | – | 0.00 | 1.00 |
| goby_battle_engine__spec feature reconstruction · goby | feature implementation f2p | hard | – | – | – | 0.00 | 1.00 |
| javamod_tracker_replay feature reconstruction · javamod | feature implementation f2p | hard | 36% k=6 | – | – | 0.00 | 1.00 |
| javamod_tracker_replay__spec feature reconstruction · javamod | feature implementation f2p | hard | 40% k=6 | – | – | 0.00 | 1.00 |
| x18xx_step_mechanics feature reconstruction · x18xx | feature implementation f2p | hard | 11% k=5 | – | – | 0.00 | 1.00 |
| x18xx_step_mechanics__spec feature reconstruction · x18xx | feature implementation f2p | hard | 12% k=5 | – | – | 0.00 | 1.00 |
| incident_crosstenant_hardened incident response · sento-flow-backend | security demonstrate then patch | hard | 9% k=8 | – | – | 0.00 | 1.00 |
| incident_crosstenant_hardened__spec incident response · sento-flow-backend | security demonstrate then patch | hard | 21% k=8 | – | – | 0.00 | 1.00 |
| incident_xsub_dsn incident response · sento-flow-backend | incident response | hard | 53% k=9 | – | – | 0.00 | 1.00 |
| incident_xsub_dsn__spec incident response · sento-flow-backend | incident response | hard | 33% k=9 | – | – | 0.00 | 1.00 |
| incident_xsub_prepared_xact incident response · sento-flow-backend | incident response | hard | 17% k=9 | – | – | 0.00 | 1.00 |
| incident_xsub_prepared_xact__spec incident response · sento-flow-backend | incident response | hard | 23% k=9 | – | – | 0.00 | 1.00 |
| incident_xsub_searchpath incident response · sento-flow-backend | incident response | hard | 41% k=9 | – | – | 0.00 | 1.00 |
| incident_xsub_searchpath__spec incident response · sento-flow-backend | incident response | hard | 40% k=9 | – | – | 0.00 | 1.00 |
| incident_xsubsystem_onboarding incident response · sento-flow-backend | incident response | hard | 24% k=9 | – | – | 0.00 | 1.00 |
| incident_xsubsystem_onboarding__spec incident response · sento-flow-backend | incident response | hard | 16% k=9 | – | – | 0.00 | 1.00 |
| async_executor_runtime_migration migration and modernization · async-artifact | constructed migration | hard | 62% k=3 | – | – | 0.00 | 1.00 |
| mocha_chai_to_node_test migration and modernization · karma-migrate | constructed migration | medium | 66% k=8 | – | – | 0.00 | 1.00 |
| mocha_chai_to_node_test__terse migration and modernization · karma-migrate | constructed migration | medium | 0% k=8 | – | – | 0.00 | 1.00 |
| quickcheck_to_proptest migration and modernization · itertools-migrate | constructed migration | medium | 2% k=8 | – | – | 0.00 | 1.00 |
| rxjava3_jdk21_multi_project_migration migration and modernization · rxjava3-composite | constructed migration | hard | 14% k=3 | – | – | 0.00 | 1.00 |
| scala3_macro_migration migration and modernization · muster-scala | constructed migration | hard | 52% k=5 | – | – | 0.00 | 1.00 |
| upgrade_rn_0_74_to_0_80 migration and modernization · lovewick-rn | long horizon upgrade | hard | – | – | – | 0.00 | 1.00 |
| battery_aging_regime_shift scientific and ml research · sciml-realdata | real data regime shift forecast | hard | – | 0% k=3 | 32% k=6 | – | – |
| bore_flux_tomography scientific and ml research · sciml-bench | scientific inverse problem | hard | – | 20% k=3 | 27% k=3 | – | – |
| coat_stack_admittance_c scientific and ml research · sciml-bench | synthetic structure identification | hard | – | 7% k=6 | 28% k=6 | – | – |
| coat_stack_admittance_f scientific and ml research · sciml-bench | synthetic structure identification | hard | – | 13% k=3 | 19% k=3 | – | – |
| coat_stack_admittance_id scientific and ml research · sciml-bench | synthetic structure identification | hard | – | 30% k=6 | 58% k=5 | – | – |
| congestion_service_id scientific and ml research · sciml-bench | synthetic structure identification | hard | – | 61% k=3 | 69% k=3 | – | – |
| dark_species_kinetics scientific and ml research · sciml-bench | synthetic structure identification | hard | – | 9% k=3 | 10% k=3 | – | – |
| diffusivity_profile_id scientific and ml research · sciml-bench | synthetic structure identification | hard | – | 22% k=3 | 24% k=3 | – | – |
| frame_hysteresis_id scientific and ml research · sciml-bench | synthetic structure identification | hard | – | 5% k=3 | 11% k=3 | – | – |
| kepler_transit_timing_injection scientific and ml research · sciml-realdata | real data injection recovery | hard | – | 16% k=3 | 17% k=3 | – | – |
| latent_basin_transfer_opt scientific and ml research · ml-method | benchmark ml method | hard | – | 12% k=3 | 32% k=3 | – | – |
| layered_contact_thermography scientific and ml research · sciml-bench | synthetic structure identification | hard | – | 15% k=3 | 15% k=3 | – | – |
| ligo_strain_injection_recovery scientific and ml research · sciml-realdata | real data injection recovery | hard | – | 10% k=3 | 14% k=3 | – | – |
| multifidelity_transfer_opt scientific and ml research · ml-method | benchmark ml method | hard | – | 2% k=3 | 1% k=3 | – | – |
| noisy_constrained_transfer_opt scientific and ml research · ml-method | benchmark ml method | hard | – | 1% k=3 | 19% k=3 | – | – |
| slab_phase_tomography scientific and ml research · sciml-bench | synthetic structure identification | hard | – | 15% k=3 | 59% k=3 | – | – |
| sparse_diffusion_reaction_reconstruction scientific and ml research · pdebench-inverse | scientific inverse pde | research | – | 28% k=3 | 41% k=3 | – | – |
| terminal_device_id scientific and ml research · sciml-bench | synthetic system identification | hard | – | 23% k=3 | 17% k=3 | – | – |
| sec_access_control_permission_system security hardening · brodboksen-backend | security access control | hard | 49% k=9 | – | – | 0.00 | 1.00 |
| sec_access_control_permission_system__terse security hardening · brodboksen-backend | security access control | hard | 37% k=9 | – | – | 0.00 | 1.00 |
| sec_access_policy_compiler security hardening · gaterail | security access control | hard | 40% k=4 | – | – | 0.00 | 1.00 |
| sec_access_policy_compiler__terse security hardening · gaterail | security access control | hard | 40% k=4 | – | – | 0.00 | 1.00 |
| sec_detection_rule_compiler security hardening · sentryql | security query construction | hard | 44% k=4 | – | – | 0.00 | 1.00 |
| sec_detection_rule_compiler__terse security hardening · sentryql | security query construction | hard | 44% k=4 | – | – | 0.00 | 1.00 |
| sec_filter_sql_query_builder security hardening · brodboksen-backend | security query construction | hard | 38% k=11 | – | – | 0.00 | 1.00 |
| sec_filter_sql_query_builder__terse security hardening · brodboksen-backend | security query construction | hard | 31% k=10 | – | – | 0.00 | 1.00 |
| sec_log_normalizer security hardening · normalog | security log normalization | hard | 33% k=4 | – | – | 0.00 | 1.00 |
| sec_log_normalizer__terse security hardening · normalog | security log normalization | hard | 25% k=4 | – | – | 0.00 | 1.00 |
| sec_waf_rule_compiler security hardening · wafforge | security query construction | hard | 40% k=4 | – | – | 0.00 | 1.00 |
| sec_waf_rule_compiler__terse security hardening · wafforge | security query construction | hard | 40% k=4 | – | – | 0.00 | 1.00 |
| cavalier_offset_kernel specialist algorithmic · cavc-geometry-kernel | feature implementation f2p | hard | 43% k=9 | – | – | 0.00 | 1.00 |
| cavalier_offset_kernel__spec specialist algorithmic · cavc-geometry-kernel | feature implementation f2p | hard | 34% k=8 | – | – | 0.00 | 1.00 |
| llnl_units_string_parser specialist algorithmic · llnl-units | feature implementation f2p | hard | 29% k=8 | – | – | 0.00 | 1.00 |
| llnl_units_string_parser__terse specialist algorithmic · llnl-units | feature implementation f2p | hard | 19% k=9 | – | – | 0.00 | 1.00 |
| retroboy_gb_apu specialist algorithmic · retroboy | feature implementation f2p | hard | 33% k=8 | – | – | 0.00 | 1.00 |
| retroboy_gb_apu__terse specialist algorithmic · retroboy | feature implementation f2p | hard | 33% k=4 | – | – | 0.00 | 1.00 |
| soundfont_synth specialist algorithmic · spessasynth | feature implementation f2p | hard | 26% k=4 | – | – | 0.00 | 1.00 |
| soundfont_synth__terse specialist algorithmic · spessasynth | feature implementation f2p | hard | 26% k=4 | – | – | 0.00 | 1.00 |
| symengine_equation_solver specialist algorithmic · symengine | feature implementation f2p | hard | 25% k=4 | – | – | 0.00 | 1.00 |
| symengine_equation_solver__terse specialist algorithmic · symengine | feature implementation f2p | hard | 9% k=8 | – | – | 0.00 | 1.00 |
| ym2612_fm_synthesis specialist algorithmic · jgenesis | feature implementation f2p | hard | 42% k=8 | – | – | 0.00 | 1.00 |
| ym2612_fm_synthesis__terse specialist algorithmic · jgenesis | feature implementation f2p | hard | 38% k=8 | – | – | 0.00 | 1.00 |
| ym2612_opn2_fm specialist algorithmic · helios | feature implementation f2p | hard | 40% k=4 | – | – | 0.00 | 1.00 |
| ym2612_opn2_fm__terse specialist algorithmic · helios | feature implementation f2p | hard | 43% k=4 | – | – | 0.00 | 1.00 |