← Back to Benchmark Results

openrouter/qwen/qwen3-max-thinking

51.8%
Pass Rate
29/56
Tasks Passed
3
Runs
47.0%
pass@1
51.8%
pass@3
89.3%
Consistency
0.1
Temperature
-
Thinking
404,678
Tokens
$2.97
Cost
1st: 502nd: 29Failed: 2729/56 passed

Known Shortcomings (12)

Sorted by occurrence count (most frequent first)

# Concept AL Concept Count Affected Tasks
1 option-field-optionmembers-required table-extension-field-properties 2 CG-AL-E006, CG-AL-M006

Description: The model generated a table extension with an Option field but left the OptionMembers property blank or omitted it entirely. In AL, when defining a field of type Option, the OptionMembers property must be explicitly set with at least one value (e.g., OptionMembers = Email,Phone,Mail,SMS). The compiler error AL0153 indicates 'The property OptionMembers cannot be blank'. The model failed to properly define the OptionMembers for the Preferred Contact Method field in the table extension.

Correct Pattern:
field(70000; "Preferred Contact Method"; Option)
{
    Caption = 'Preferred Contact Method';
    OptionMembers = Email,Phone,Mail,SMS;
    OptionCaption = 'Email,Phone,Mail,SMS';
}
Incorrect Pattern:
field(70000; "Preferred Contact Method"; Option)
{
    Caption = 'Preferred Contact Method';
    OptionMembers = ;
}

Error Codes: AL0153, AL0104

2 enum-frominteger-syntax enum-handling 1 CG-AL-H007

Description: The model failed to generate valid code entirely (generated code not found / compilation failed). The AL0151 error 'Expression must be an Option' at line 18:22 indicates the model likely used Enum.FromInteger() or similar enum conversion incorrectly, or passed an enum value where an Option was expected. The model did not understand how to properly work with ErrorInfo.CustomDimensions and enum integer conversion in modern AL. The task requires using Format() to convert enum ordinal to text for CustomDimensions, and Enum::"CG Validation Error".FromInteger() to convert back, along with ErrorInfo.Create() pattern. The model either failed to produce code at all or produced code with fundamental enum/ErrorInfo handling errors.

Correct Pattern:
To convert enum to integer for CustomDimensions: Format(ErrorCode.AsInteger()). To convert back: Enum::"CG Validation Error".FromInteger(IntValue). ErrorInfo should be created via ErrorInfo.Create() and properties set individually: ErrInfo.Message(ErrorMessage); ErrInfo.CustomDimensions.Add('ErrorCode', Format(ErrorCode.AsInteger())); ErrInfo.CustomDimensions.Add('FieldName', FieldName); ErrInfo.Collectible(IsCollectingErrors());
Incorrect Pattern:
// Generated code not found - but error at line 18:22 suggests incorrect enum usage, likely passing an Enum where an Option/Integer was expected

Error Codes: AL0151

3 list-iteration-pattern list-of-type-operations 1 CG-AL-M002

Description: The model attempted to use a 'Sum' method on 'List of [Decimal]' which does not exist in AL. In AL, the List type does not have a Sum() method. To sum elements of a List of [Decimal], you must iterate through the list using a foreach loop or index-based for loop and accumulate the total manually. The model failed to generate valid code for the CalculateOrderTotal procedure, using a non-existent .Sum method instead of iterating over the list.

Correct Pattern:
procedure CalculateOrderTotal(LineTotals: List of [Decimal]): Decimal
var
    LineTotal: Decimal;
    Total: Decimal;
begin
    Total := 0;
    foreach LineTotal in LineTotals do
        Total += LineTotal;
    exit(Total);
end;
Incorrect Pattern:
LineTotals.Sum

Error Codes: AL0132

4 variant-type-argument-and-interface-definition interface-definition 1 CG-AL-H023

Description: The model made multiple errors: (1) It tried to use 'Variant' as a type argument in a generic collection (e.g., Dictionary of [Text, Variant] or similar), but AL does not allow Variant as a type argument in generic types like Dictionary or List. (2) It failed to create the 'IFieldTransformer' interface as a separate AL interface object, instead likely trying to reference it as a Codeunit. In AL, interfaces are defined with 'interface "IFieldTransformer"' syntax and are distinct from codeunits. The TransformRecord procedure should accept a parameter implementing the interface, not a Codeunit type directly referencing the interface name as a codeunit.

Correct Pattern:
1) Avoid using Variant as a type argument - use Text or specific types instead. 2) Define a separate interface object: 'interface "IFieldTransformer" { procedure Transform(var FRef: FieldRef): Variant; }' and reference it in the procedure signature as 'Transformer: Interface "IFieldTransformer"' rather than 'Codeunit "IFieldTransformer"'.
Incorrect Pattern:
Lines using Variant as type argument (e.g., Dictionary of [Text, Variant]) and referencing 'Codeunit IFieldTransformer' instead of properly defining and using an interface

Error Codes: AL0408

5 json-object-api-methods json-handling 1 CG-AL-M005

Description: The model generated code using incorrect AL JSON API methods. Multiple errors show fundamental misunderstanding of the AL JSON types: (1) JsonObject doesn't have a 'Clear' method - should use Clear(variable) built-in or reinitialize, (2) 'JSON Management' codeunit doesn't have 'TryGetObject' method, (3) JsonToken doesn't have 'IsText' or 'AsText' methods - should use Token.IsValue() and Token.AsValue().AsText(), (4) JsonObject doesn't have a 'Count' method - should use Keys.Count or similar pattern. The model confused AL's native JSON types (JsonObject, JsonToken, JsonValue) API with other frameworks or older BC JSON patterns.

Correct Pattern:
Use Clear(JsonObjectVar) for clearing, JsonObject.Get('key', Token) for retrieval, Token.IsValue() and Token.AsValue().AsText() for type checking and value extraction, and JsonObject.Keys.Count for counting properties. Use JsonObject.Contains('key') to check field existence.
Incorrect Pattern:
JsonObject.Clear(); JSONManagement.TryGetObject(); JsonToken.IsText(); JsonToken.AsText(); JsonObject.Count()

Error Codes: AL0132

6 bc-event-names-and-table-references event-subscribers-and-bc-object-model 1 CG-AL-M008

Description: The model generated code with multiple fundamental errors about the BC object model: (1) It referenced non-existent events 'OnAfterInsert' and 'OnAfterModify' on the Purchase Header table - the correct integration events have different names (e.g., 'OnAfterInsertEvent' or the model should use different event patterns). (2) It referenced non-existent tables 'Workflow Setup' and 'Workflow Comment Line' - these are not standard BC tables. (3) It referenced a non-existent codeunit 'SMTP Mail' which was deprecated/removed in modern BC versions. The model lacks knowledge of the actual BC event naming conventions, the correct BC workflow-related table names, and the current email API in Business Central.

Correct Pattern:
For a self-contained implementation: avoid subscribing to non-existent events, use in-memory tracking (Dictionary or temporary records) instead of referencing non-existent tables like 'Workflow Setup' and 'Workflow Comment Line', and use the modern Email module (Codeunit "Email Message"/"Email") instead of the deprecated 'SMTP Mail'. For event subscribers on Purchase Header, use actual published events like 'OnAfterValidateEvent' or integration events that actually exist on the table.
Incorrect Pattern:
EventSubscriber(ObjectType::Table, Database::"Purchase Header", 'OnAfterInsert', ...)
EventSubscriber(ObjectType::Table, Database::"Purchase Header", 'OnAfterModify', ...)
Table 'Workflow Setup'
Table 'Workflow Comment Line'
Codeunit 'SMTP Mail'

Error Codes: AL0280

7 report-dataitem-trigger-syntax report-definition 1 CG-AL-M007

Description: The model generated a report with multiple syntax errors including incorrect brace placement in dataitem definitions, and used 'OnPostDataItem' which is not a valid trigger in AL reports. The valid triggers for report dataitems are OnPreDataItem, OnAfterGetRecord, and OnPostDataItem is not recognized. The compilation errors (AL0104 for missing '}' and AL0162 for invalid trigger) indicate the model does not properly understand AL report structure with nested dataitems, proper trigger names, and correct brace/block scoping. The model also likely had structural issues with how dataitem blocks are nested and closed, leading to cascading syntax errors.

Correct Pattern:
report 70001 "Sales Performance Analysis"
{
    UsageCategory = ReportsAndAnalysis;
    ApplicationArea = All;
    DefaultRenderingLayout = RDLCLayout;

    dataset
    {
        dataitem(Customer; Customer)
        {
            RequestFilterFields = "No.", "Customer Posting Group";
            column(CustomerNo; "No.") { }
            column(CustomerName; Name) { }

            trigger OnPreDataItem()
            begin
                // initialization
            end;

            trigger OnAfterGetRecord()
            begin
                // calculations
            end;

            dataitem(SalesHeader; "Sales Header")
            {
                DataItemLink = "Sell-to Customer No." = field("No.");
                DataItemTableView = where("Document Type" = const(Order));

                dataitem(SalesLine; "Sales Line")
                {
                    DataItemLink = "Document No." = field("No."), "Document Type" = field("Document Type");

                    trigger OnAfterGetRecord()
                    begin
                        // line calculations
                    end;
                }
            }
        }
    }

    requestpage
    {
        layout
        {
            area(Content)
            {
                group(Options)
                {
                    // filter fields
                }
            }
        }
    }

    rendering
    {
        layout(RDLCLayout)
        {
            Type = RDLC;
            LayoutFile = 'SalesPerformanceAnalysis.rdl';
        }
    }
}
Incorrect Pattern:
Generated code not available but errors indicate: mismatched braces at lines 43, 76, 113; use of 'OnPostDataItem' trigger at line 113; and structural collapse at line 120

Error Codes: AL0104

8 jsonobject-get-selecttoken-pattern json-api-usage 1 CG-AL-M020

Description: The model incorrectly used JsonObject.Get() by passing typed variables (Text, Decimal, Boolean, Integer, JsonArray) directly as the second argument, but JsonObject.Get() returns a JsonToken via a 'var JsonToken' parameter. The model also tried to use non-existent methods like JsonToken.IsInteger() and JsonToken.AsInteger(). The correct pattern is to first get a JsonToken, then convert it to a JsonValue, and use AsText(), AsDecimal(), AsBoolean(), AsInteger() on the JsonValue. For arrays, you get the JsonToken and then use AsArray().

Correct Pattern:
var JToken: JsonToken; JValue: JsonValue;
ProductJson.Get('name', JToken);
JValue := JToken.AsValue();
NameValue := JValue.AsText();
// For arrays:
ProductJson.Get('values', JToken);
ValuesArray := JToken.AsArray();
// For array elements:
ValuesArray.Get(i, JToken);
JValue := JToken.AsValue();
IntVal := JValue.AsInteger();
Incorrect Pattern:
ProductJson.Get('name', NameValue); // where NameValue is Text
ProductJson.Get('price', PriceValue); // where PriceValue is Decimal
JsonToken.IsInteger()
JsonToken.AsInteger()

Error Codes: AL0133

9 flowfield-calcformula-syntax flowfield 1 CG-AL-M010

Description: The model generated a FlowField (likely for 'Actual Cost') with an incorrect CalcFormula syntax. The AL compiler error AL0176 at line 125 indicates that the model used an invalid calculation formula method. In AL, CalcFormula must use one of the recognized methods: Average, Count, Exist, Min, Max, Lookup, or Sum. The model likely wrote something syntactically invalid such as a malformed Sum expression or used an unsupported method/syntax in the CalcFormula property. Additionally, the task asked for 'Actual Cost' to be 'calculated from tasks', which should use a FlowField with CalcFormula = Sum("Project Task"."some field" WHERE("Project Code" = FIELD("Project Code"))), but the model got the syntax wrong.

Correct Pattern:
field(6; "Actual Cost"; Decimal)
{
    FieldClass = FlowField;
    CalcFormula = Sum("Project Task"."Actual Hours" WHERE("Project Code" = FIELD("Project Code")));
    // Or alternatively sum of (Actual Hours * Hourly Rate) but that requires an intermediate field
}
Incorrect Pattern:
The generated code at line 125 contained an invalid CalcFormula expression that did not use one of the recognized formula methods (Average, Count, Exist, Min, Max, Lookup, Sum)

Error Codes: AL0176

10 yaml-parsing-variable-naming-and-syntax al-syntax-reserved-words-and-json-iteration 1 CG-AL-M021

Description: The model generated AL code with multiple syntax errors. The primary issue at line 84 is using 'Key' as a variable name or in an invalid context - 'Key' may be treated as a reserved word or the model used incorrect syntax for iterating over JsonObject keys. Additional errors suggest malformed expressions and mismatched parentheses/blocks. The model failed to properly implement YAML parsing (which requires manual string splitting in AL since there's no native YAML library) and JSON object iteration patterns. The model likely tried to use foreach-style iteration over JsonObject keys using incorrect syntax, and may have used reserved words as variable identifiers without proper quoting.

Correct Pattern:
In AL, to iterate over JsonObject members, use JsonObject.Keys() which returns a List of Text. Declare a variable like 'KeyName: Text' (avoiding reserved words), get keys via JObj.Keys, then use foreach KeyName in KeysList. For YAML parsing, manually split text by line feed characters and parse 'key: value' pairs using StrPos/CopyStr or TextSplit. Example: KeysList := JObj.Keys; foreach KeyName in KeysList do begin JObj.Get(KeyName, JToken); ... end;
Incorrect Pattern:
Key (line 84 context - likely something like 'foreach Key in JsonObj.Keys do' or similar invalid pattern)

Error Codes: AL0519

11 temporary-table-parameter-handling temporary-table 1 CG-AL-H003

Description: The test TestHighInventoryDiscount fails because the model's generated code does not correctly populate the TempResult temporary record with items that have inventory >= 100. The test finds an Item with Inventory >= 100 and Unit Price > 0, then calls ProcessItemsWithDiscount with MinDiscount=15, and expects to find that item in TempResult with Discount Percent = 15. The failure 'Assert.IsTrue failed. High inventory item should be in results' indicates the item was not inserted into TempResult. This is likely because the model's implementation has a bug in how it loops through items, assigns line numbers, calculates discounts, or inserts records into the temporary table. Since the generated code was not captured ('Generated code not found'), but the app compiled and other tests passed (only TestHighInventoryDiscount failed), the model likely had an issue with the discount tier logic (e.g., using wrong comparison operators like '>' instead of '>=' for the inventory thresholds), causing items with exactly 100 inventory to get 10% instead of 15%, thus being excluded when MinDiscount=15.

Correct Pattern:
if Item.Inventory >= 100 then
    DiscountPercent := 15
else if Item.Inventory >= 50 then
    DiscountPercent := 10
else if Item.Inventory >= 10 then
    DiscountPercent := 5
else
    DiscountPercent := 0;

Note: Must use >= (greater than or equal) not > (greater than) for inventory thresholds.
Incorrect Pattern:
// Generated code not found - but the compiled app had a logic error in discount tier calculation
12 word-counting-with-extra-spaces string-manipulation 1 CG-AL-E005

Description: The model's CountWords implementation does not correctly handle extra/leading/trailing spaces. The test 'TestCountWordsExtraSpaces' passes ' hello world ' and expects 2 words, but the model's implementation returned 8, indicating it likely counted individual characters or spaces rather than properly splitting by whitespace and ignoring empty segments. A correct implementation needs to iterate through the string, tracking transitions from space to non-space characters (or splitting and filtering empty entries).

Correct Pattern:
procedure CountWords(InputText: Text): Integer
var
    i: Integer;
    WordCount: Integer;
    InWord: Boolean;
begin
    if InputText = '' then
        exit(0);
    InWord := false;
    WordCount := 0;
    for i := 1 to StrLen(InputText) do begin
        if InputText[i] <> ' ' then begin
            if not InWord then begin
                WordCount += 1;
                InWord := true;
            end;
        end else
            InWord := false;
    end;
    exit(WordCount);
end;
Incorrect Pattern:
// Generated code not available, but the CountWords procedure returned 8 for input '  hello   world  ' instead of 2