← Back to Benchmark Results

anthropic/claude-opus-4-6

76.8%
Pass Rate
43/56
Tasks Passed
3
Runs
76.2%
pass@1
76.8%
pass@3
98.2%
Consistency
0.1
Temperature
-
Thinking
650,237
Tokens
$11.17
Cost
1st: 1052nd: 23Failed: 1343/56 passed

Known Shortcomings (6)

Sorted by occurrence count (most frequent first)

# Concept AL Concept Count Affected Tasks
1 reserved-keyword-as-parameter-name al-syntax-reserved-words 1 CG-AL-H014

Description: The model used 'Key' as a parameter name in the SafeGetText procedure. In AL, 'key' is a reserved keyword and cannot be used as an identifier. The model should have used a non-reserved name like 'KeyName', 'PropertyKey', or quoted it with double quotes. The task prompt specified 'Text named Key' which is itself problematic since 'Key' is reserved, but the model should have known to escape or rename it.

Correct Pattern:
procedure SafeGetText(Obj: JsonObject; "Key": Text; DefaultValue: Text): Text
// Or use a non-reserved name like KeyName:
procedure SafeGetText(Obj: JsonObject; KeyName: Text; DefaultValue: Text): Text
Incorrect Pattern:
procedure SafeGetText(Obj: JsonObject; Key: Text; DefaultValue: Text): Text

Error Codes: AL0105

2 cross-join-dataitem-link query-dataitem-joins 1 CG-AL-H017

Description: The model incorrectly added a DataItemLink on the 'DimensionSetEntry' dataitem that references a field from the 'Department' dataitem (DataItemLink = "Dimension Code" = Department."Dimension Code"). In AL queries, a Column or Filter source must reference a field defined on the table of its parent DataItem. With CrossJoin between Department and Project, the DimensionSetEntry is nested under Project, so its DataItemLink can only reference fields from its direct parent (Project) or use fields from its own table. The reference to Department."Dimension Code" is invalid because 'Dimension Code' on the Dimension Value table (Department) is not a field on the Dimension Set Entry table's parent chain in the way the compiler expects. Additionally, for a LeftOuterJoin, the DataItemLink must reference the immediate parent dataitem's fields. The model should have either linked DimensionSetEntry to Project's fields or used an appropriate link structure that the compiler accepts.

Correct Pattern:
dataitem(DimensionSetEntry; "Dimension Set Entry")
{
    SqlJoinType = LeftOuterJoin;
    DataItemLink = "Dimension Code" = Project."Dimension Code", "Dimension Value Code" = Project."Code";

    column(MatchCount; "Entry No.")
    {
        Method = Count;
    }
}
Incorrect Pattern:
dataitem(DimensionSetEntry; "Dimension Set Entry")
{
    SqlJoinType = LeftOuterJoin;
    DataItemLink = "Dimension Code" = Department."Dimension Code";

    column(MatchCount; "Entry No.")
    {
        Method = Count;
    }
}

Error Codes: AL0345

3 incomplete-procedure-body code-generation-completeness 1 CG-AL-M009

Description: The model generated a truncated/incomplete procedure body for the 'ContainsDigit' local procedure. The code cuts off at 'var\n i: Integer;\n Current' without completing the variable declaration, procedure body, or closing the codeunit. This caused cascading syntax errors. The model failed to produce complete code, likely due to output token limits or generation issues, resulting in an unterminated procedure and codeunit block.

Correct Pattern:
local procedure ContainsDigit(InputText: Text): Boolean
    var
        i: Integer;
        CurrentChar: Char;
    begin
        for i := 1 to StrLen(InputText) do begin
            CurrentChar := InputText[i];
            if (CurrentChar >= '0') and (CurrentChar <= '9') then
                exit(true);
        end;
        exit(false);
    end;
Incorrect Pattern:
local procedure ContainsDigit(InputText: Text): Boolean
    var
        i: Integer;
        Current

Error Codes: AL0104

4 flowfield-calcfields-requirement flowfield 1 CG-AL-H003

Description: The Item.Inventory field is a FlowField in Business Central. FlowFields are not automatically calculated when reading records - you must explicitly call CalcFields before accessing their value. The generated code reads Item.Inventory directly without calling Item.CalcFields(Inventory), so the value is always 0, resulting in 0% discount for all items. The test finds items with Inventory >= 100 (using SetFilter which evaluates at SQL level) and expects them to appear with 15% discount, but they don't because the codeunit sees Inventory as 0.

Correct Pattern:
Item.CalcFields(Inventory);
DiscountPercent := CalculateDiscount(Item.Inventory);
Incorrect Pattern:
DiscountPercent := CalculateDiscount(Item.Inventory);
5 parse-failure unknown 1 CG-AL-M005

Description: Failed to parse LLM analysis response: { "outcome": "model_shortcoming", "category": "model_knowledge_gap", "concept": "multi-file-output-formatting", "alConcept": "code-generation-output-format", "description": "The model includ

Correct Pattern:
Incorrect Pattern:
6 report-labels-block-closure report-structure 1 CG-AL-M007

Description: The model failed to properly close the report object. The generated code has an incomplete 'labels' block at the end of the report - it starts the labels section but never closes it with proper closing braces. Additionally, the model concatenated two separate AL files into one file, placing the codeunit 80097 code directly after the incomplete report definition. The labels block is missing closing braces for the labels section, the report object itself, and then the second file (the mock calculator codeunit) is appended without proper file separation. This results in unexpected characters (backticks from markdown formatting) at line 453 and missing structural elements. The model also failed to include several helper procedures referenced in the report (BuildCustomerRankings, BuildTopProductsList, CalcInvoiceAmounts, CalcInvoiceQuantity, CalcCustomerSalesAmount, CalcCustomerSalesQty, CalcCustomerOrderCount, CalcRegionTotals, IsItemTopProduct, GetCustomerRank) and the required variable declarations.

Correct Pattern:
The labels block should be properly closed with '}', followed by variable declarations (var section) with all referenced variables, all helper procedure implementations, and then the report closing '}'. The mock calculator codeunit should be in a separate file. Example:

    labels
    {
        CustomerNoLbl = 'Customer No.';
        // ... more labels
    }

    var
        StartDate: Date;
        EndDate: Date;
        // ... all other variables

    // ... all helper procedures
}
Incorrect Pattern:
    labels
    {
        CustomerNoLbl = 'Customer No.';
        CustomerNameLbl = 'Customer Name';
        RegionLbl = 'Region';
        CategoryLbl = 'Category';
        TotalSalesLbl = 'Total Sales';
        OrderCountLbl = 'Order Count';
        AvgOrderValueLbl = 'Avg. Order Value';
        YoYGrowthLbl = 'YoY Growth %';

// File: CG-AL-M007.MockCalculator.al
codeunit 80097 "CG-AL-M007 Mock Calculator"

Error Codes: AL0183