← Back to Benchmark Results

openai/gpt-5.2-2025-12-11@thinking=high

69.6%
Pass Rate
39/56
Tasks Passed
3
Runs
64.3%
pass@1
69.6%
pass@3
89.3%
Consistency
0.1
Temperature
high
Thinking
419,417
Tokens
$3.63
Cost
1st: 772nd: 31Failed: 1739/56 passed

Known Shortcomings (10)

Sorted by occurrence count (most frequent first)

# Concept AL Concept Count Affected Tasks
1 interface-definition-syntax interface-definition 2 CG-AL-E008, CG-AL-M009

Description: The model failed to generate any valid AL code. The generated code was either empty or completely malformed (noted as 'Generated code not found'). The compilation errors (AL0107, AL0104, AL0198) indicate the output file existed but contained invalid content that doesn't start with a recognized AL object keyword. The model needed to generate an AL interface definition, which in Business Central AL uses the syntax 'interface "Payment Processor" { ... }' without an ID number (unlike codeunits/tables). The task description incorrectly mentions 'ID 70000' for the interface, but AL interfaces do not have IDs - the model should have known to omit it. Additionally, the test file references a mock codeunit 'CG-AL-E008 Mock Processor' that would also need to be generated. The fundamental failure is that the model produced no usable code at all.

Correct Pattern:
interface "Payment Processor"
{
    /// <summary>
    /// Processes a payment for the given amount and payment method.
    /// </summary>
    procedure ProcessPayment(Amount: Decimal; PaymentMethod: Text): Boolean;

    /// <summary>
    /// Validates the provided payment data.
    /// </summary>
    procedure ValidatePayment(PaymentData: Text): Boolean;

    /// <summary>
    /// Gets the transaction fee for the given amount.
    /// </summary>
    procedure GetTransactionFee(Amount: Decimal): Decimal;
}
Incorrect Pattern:
// Generated code not found

Error Codes: AL0107

2 table-field-caption-property table-definition 1 CG-AL-H003

Description: The model generated a table definition for 'CG Discount Result' (ID 70203) where it incorrectly used the 'Caption' property in a context where it is not allowed (AL0124 error at line 47). This suggests the model placed a Caption property on a field or object element where AL does not permit it, or used incorrect syntax for table field definitions. The generated code was not captured in the output, but the compilation error clearly indicates the model produced invalid AL code with a misplaced Caption property. This is a model knowledge gap about proper AL table definition syntax and where the Caption property can and cannot be used.

Correct Pattern:
In AL table definitions, Caption property is valid on table objects and field definitions but not on all elements. For fields, the syntax is: field(1; "Line No."; Integer) { Caption = 'Line No.'; } - ensure Caption is only used within valid property contexts.
Incorrect Pattern:
Caption = '...'; // placed in an invalid context (line 47 of generated file)

Error Codes: AL0124

3 query-object-syntax query-definition 1 CG-AL-H011

Description: The model failed to generate valid AL code for a query object. The generated code has syntax errors at line 31, indicating the model does not properly understand the structure of an AL query object, including how to define dataitems, columns with aggregation methods (Sum, Count), filter elements with ColumnFilter, and OrderBy properties. The 'Generated code not found' note and the compilation errors (expecting '}' and application object keywords) suggest the model either produced incomplete/malformed code or failed to produce code entirely.

Correct Pattern:
query 70011 "CG Sales Summary"
{
    QueryType = Normal;
    OrderBy = ascending(Sell_to_Customer_No);

    elements
    {
        dataitem(SalesLine; "Sales Line")
        {
            column(Document_No; "Document No.") { }
            column(Sell_to_Customer_No; "Sell-to Customer No.") { }
            column(Line_Amount_Sum; "Line Amount") { Method = Sum; }
            column(Line_Count; "Line Amount") { Method = Count; }
            filter(Document_Type; "Document Type") { ColumnFilter = const(Order); }
        }
    }
}
Incorrect Pattern:
// Generated code not found - the model produced syntactically invalid AL query code that failed at line 31

Error Codes: AL0104

4 query-crossjoin-syntax query-object-definition 1 CG-AL-H017

Description: The model failed to generate valid AL code for a query object with CrossJoin. The compilation error at line 29:57 ('identifier expected') suggests the model produced syntactically invalid AL code, likely mishandling the CrossJoin dataitem structure, column references, or the SqlJoinType property syntax for query objects. The generated code was not captured ('Generated code not found'), but the error pattern indicates the model doesn't properly understand how to construct AL query objects with CrossJoin between two instances of the same table ('Dimension Value'), including proper dataitem naming, SqlJoinType assignment, and column definitions with Method = Count.

Correct Pattern:
query 70017 "CG Dimension Matrix"
{
    QueryType = Normal;

    elements
    {
        dataitem(DeptDimValue; "Dimension Value")
        {
            SqlJoinType = CrossJoin;
            DataItemTableFilter = "Dimension Code" = const('DEPARTMENT');
            column(DepartmentCode; "Code") { }
            column(DepartmentName; Name) { }

            dataitem(ProjDimValue; "Dimension Value")
            {
                SqlJoinType = CrossJoin;
                DataItemTableFilter = "Dimension Code" = const('PROJECT');
                column(ProjectCode; "Code") { }
                column(ProjectName; Name) { }

                dataitem(DimSetEntry; "Dimension Set Entry")
                {
                    SqlJoinType = LeftOuterJoin;
                    DataItemLink = "Dimension Code" = DeptDimValue."Dimension Code";
                    column(MatchCount; "Dimension Value Code")
                    {
                        Method = Count;
                    }
                }
            }
        }
    }
}
Incorrect Pattern:
// Generated code not found - but compilation failed at line 29:57 with syntax error

Error Codes: AL0107

5 parse-failure unknown 1 CG-AL-H014

Description: Failed to parse LLM analysis response: Looking at this failure, I need to understand what happened: 1. The task asks the model to create a codeunit that uses "typed JSON getter methods" like `GetText('name')`, `GetInteger('age')`, `GetBoo

Correct Pattern:
Incorrect Pattern:
6 codeunit-self-reference fluent-api-pattern 1 CG-AL-H018

Description: The model attempted to use 'Self' to return the current codeunit instance for fluent chaining, but 'Self' does not exist in AL. In AL, to implement a fluent API pattern where methods return the current codeunit instance, you need to use 'this' (available in newer AL versions) or pass the codeunit variable explicitly. The correct approach in AL is to declare a local variable of the codeunit type, use 'CurrCodeunit' or simply 'exit(this)' depending on the AL runtime version. A common pattern is to declare a return variable of type Codeunit and assign it via the implicit return variable or use 'exit(this)'.

Correct Pattern:
In AL, use 'exit(this);' to return the current codeunit instance for fluent API chaining. For example:

procedure SetUrl(Url: Text): Codeunit "CG Request Builder"
begin
    RequestUrl := Url;
    exit(this);
end;
Incorrect Pattern:
exit(Self);

Error Codes: AL0118

7 recordref-fieldref-dynamic-manipulation RecordRef-FieldRef-usage 1 CG-AL-H022

Description: The model failed to generate any valid code (generated code not found), and the compilation error AL0133 at line 66:50 indicates that in whatever code was produced, a Text value was passed where an Integer was expected. This is likely in the CopyMatchingFields procedure where the model attempted to look up a destination field by name (Text) using a method that expects an Integer (field number). The model lacks understanding of how to properly iterate fields using FieldIndex and match fields by name between two RecordRefs. The task requires advanced RecordRef/FieldRef patterns including FieldIndex iteration, field name matching across different RecordRefs, error handling with TryFunction patterns, and proper use of Relation property - all of which the model failed to implement correctly.

Correct Pattern:
To match fields by name between RecordRefs, iterate destination fields separately:
for i := 1 to DestRecRef.FieldCount do begin
    DestFieldRef := DestRecRef.FieldIndex(i);
    if DestFieldRef.Name = SourceFieldRef.Name then begin
        DestFieldRef.Value := SourceFieldRef.Value;
        CopiedCount += 1;
    end;
end;
Incorrect Pattern:
// The exact generated code was not captured, but the error at line 66:50 suggests something like: DestFieldRef := DestRecRef.Field(SourceFieldRef.Name) where Name (Text) is passed to Field() which expects Integer

Error Codes: AL0133

8 jsonvalue-type-checking-methods json-api-usage 1 CG-AL-M021

Description: The model attempted to use 'IsBoolean', 'IsInteger', and 'IsDecimal' methods on JsonValue, which do not exist in the AL JsonValue type. In AL's JSON API, JsonValue does not expose type-checking methods like IsBoolean, IsInteger, or IsDecimal. Instead, the model should use TryGetValue pattern or attempt to read the value as a specific type using AsBoolean(), AsInteger(), AsDecimal() wrapped in appropriate error handling, or inspect the text representation of the value to determine its type.

Correct Pattern:
Use JsonValue.AsText() to get the string representation, then determine the type by attempting conversions with Evaluate() or checking the text content (e.g., 'true'/'false' for boolean, numeric patterns for integers/decimals). Alternatively, use the AsBoolean(), AsInteger(), AsDecimal() methods inside if-then with error handling since JsonValue supports AsXxx() but not IsXxx() type-check methods.
Incorrect Pattern:
JsonValue.IsBoolean / JsonValue.IsInteger / JsonValue.IsDecimal

Error Codes: AL0132

9 report-dataitem-trigger-syntax report-definition 1 CG-AL-M007

Description: The model generated a report with multiple syntax errors including mismatched braces, incorrect trigger names (OnPostDataItem is not a valid trigger in AL reports), and malformed report structure. The compilation errors (AL0104 for missing '}', AL0162 for invalid trigger 'OnPostDataItem', AL0198 for unexpected tokens) indicate the model does not properly understand AL report object syntax, including the correct nesting of dataitem blocks, valid trigger names (OnPreDataItem, OnAfterGetRecord, OnPostDataItem is NOT valid - it should be part of the dataitem closing), and proper brace matching. The task also required a companion codeunit 'CG-AL-M007 Mock Calculator' which the model likely failed to generate or generated incorrectly. The model struggled with the complex nested structure of AL report objects with multiple data items, triggers, and request pages.

Correct Pattern:
report 70001 "Sales Performance Analysis"
{
    UsageCategory = ReportsAndAnalysis;
    ApplicationArea = All;
    DefaultRenderingLayout = RDLCLayout;

    dataset
    {
        dataitem(Customer; Customer)
        {
            // columns and nested dataitems
            trigger OnPreDataItem()
            begin
                // initialization
            end;
            trigger OnAfterGetRecord()
            begin
                // calculations
            end;
            dataitem(SalesHeader; "Sales Header")
            {
                DataItemLink = "Sell-to Customer No." = field("No.");
                dataitem(SalesLine; "Sales Line")
                {
                    DataItemLink = "Document No." = field("No.");
                }
            }
        }
    }
    requestpage { ... }
    rendering { layout(RDLCLayout) { ... } }
}

Plus a separate codeunit for 'CG-AL-M007 Mock Calculator' implementing Initialize, AddSalesLine, GetRunningTotalByCustomer, GetRunningTotalByRegion, CalculateAverageOrderValue, GetCustomerRank, GetTopProduct, GetProductSalesQuantity, CalculateYoYComparison, CalculateOrderFrequency, GetTotalSales, GetCustomerCount procedures.
Incorrect Pattern:
Report structure with OnPostDataItem trigger and mismatched braces (exact generated code not available but errors indicate structural issues at lines 85, 126, 177, 184)

Error Codes: AL0104

10 json-object-api-methods json-handling 1 CG-AL-M005

Description: The model generated AL code with multiple incorrect JSON API usages: (1) Using 'JsonObject' as a constructor call like 'JsonObject()' instead of declaring a variable of type JsonObject, (2) Calling '.ToString()' on JsonObject instead of using '.WriteTo(Text)', (3) Calling '.IsNumber' on JsonValue which doesn't exist in AL's JSON API - instead you check the value type differently or just attempt to read it. These are fundamental misunderstandings of AL's JSON type system and available methods.

Correct Pattern:
Declare JsonObject as a variable (var MyJson: JsonObject), use MyJson.WriteTo(TextVar) instead of ToString(), and for type checking use TryGetValue patterns or attempt AsDecimal() with error handling instead of IsNumber
Incorrect Pattern:
JsonObject (used as constructor), JsonObject.ToString(), JsonValue.IsNumber

Error Codes: AL0118