← Back to Benchmark Results

openrouter/z-ai/glm-5

57.1%
Pass Rate
32/56
Tasks Passed
3
Runs
45.2%
pass@1
57.1%
pass@3
75.0%
Consistency
0.1
Temperature
-
Thinking
776,176
Tokens
$6.84
Cost
1st: 522nd: 24Failed: 2432/56 passed

Known Shortcomings (17)

Sorted by occurrence count (most frequent first)

# Concept AL Concept Count Affected Tasks
1 list-dictionary-of-interface-clear-method interface-collections 1 CG-AL-H021

Description: The model generated code that calls .Clear() on List of [Interface ...] and Dictionary of [Text, Interface ...] variables. In AL, the List and Dictionary types that hold interface values do not expose a .Clear() method. The correct approach to clear these collections is to reassign them to fresh/empty instances, e.g., by declaring new local variables or by using Clear() as a built-in function (Clear(varName)) rather than as a method call on the variable. The model lacked knowledge about how to properly clear collections of interface types in AL.

Correct Pattern:
Use the built-in Clear() function instead of calling .Clear() as a method: Clear(ChannelList); Clear(ChannelDict);
Incorrect Pattern:
ChannelList.Clear();
ChannelDict.Clear();

Error Codes: AL0132

2 event-subscriber-event-name event-subscription 1 CG-AL-E010

Description: The model used the wrong event name 'OnAfterInsert' when subscribing to the Item table's insert event. In Business Central AL, the correct event name for table record insertion is 'OnAfterInsertEvent', not 'OnAfterInsert'. The compiler error AL0280 confirms that 'OnAfterInsert' is not found as an event on the Item table. The model lacks knowledge of the correct BC integration event naming convention for table triggers, which follow the pattern 'On{Before|After}{Insert|Modify|Delete|Rename}Event'.

Correct Pattern:
[EventSubscriber(ObjectType::Table, Database::Item, 'OnAfterInsertEvent', '', false, false)]
local procedure OnAfterInsertItem(var Rec: Record Item; RunTrigger: Boolean)
begin
    Message('New item created: %1', Rec."No.");
end;
Incorrect Pattern:
[EventSubscriber(ObjectType::Table, Database::Item, 'OnAfterInsert', '', false, false)]

Error Codes: AL0280

3 al-string-literal-escaping string-handling 1 CG-AL-H014

Description: The model generated code with an improperly terminated text literal on line 12. In AL, single quotes within string literals must be escaped by doubling them (''). The model likely produced a format string like 'Name: %1, Age: %2, Active: %3' or used StrSubstNo with improperly escaped quotes, or used a string concatenation that broke the text literal. The AL0360 error 'Text literal was not properly terminated' indicates the model didn't correctly handle AL string syntax, causing a cascade of syntax errors.

Correct Pattern:
In AL, text literals use single quotes and any embedded single quote must be doubled. For example: exit('Name: ' + Name + ', Age: ' + Format(Age) + ', Active: ' + Format(Active)); or use StrSubstNo('Name: %1, Age: %2, Active: %3', Name, Age, Active);
Incorrect Pattern:
Line 12 contains an improperly terminated text literal (exact generated code not available but error is on line 12 of the generated .al file)

Error Codes: AL0360

4 query-object-syntax query-definition 1 CG-AL-H011

Description: The model failed to generate valid AL code for a query object. The compilation errors (AL0107 'identifier expected' at line 13:32, AL0104 '=' expected, etc.) indicate the model produced syntactically invalid AL for the query object definition. The 'Generated code not found' note and the nature of the errors (syntax errors in the generated .al file, not the test file) suggest the model either failed to produce output or produced malformed query object syntax. AL query objects have a specific structure with dataitems, columns, and filter elements that the model did not correctly implement.

Correct Pattern:
query 70011 "CG Sales Summary"
{
    QueryType = Normal;
    OrderBy = ascending(Sell_to_Customer_No);

    elements
    {
        dataitem(SalesLine; "Sales Line")
        {
            column(Document_No; "Document No.") { }
            column(Sell_to_Customer_No; "Sell-to Customer No.") { }
            column(Line_Amount_Sum; "Line Amount") { Method = Sum; }
            column(Line_Count; "Line Amount") { Method = Count; }
            filter(Document_Type; "Document Type") { ColumnFilter = const(Order); }
        }
    }
}
Incorrect Pattern:
// Generated code not found - the model produced syntactically invalid query object code

Error Codes: AL0107

5 fluent-api-return-self-codeunit codeunit-self-reference 1 CG-AL-H018

Description: The model attempted to use 'Self' to return the current codeunit instance for fluent API chaining, but 'Self' is not a valid keyword in AL. In AL, to implement a fluent API pattern where methods return the current codeunit instance, you need to use a different approach. Since codeunits are value types in AL, the typical pattern is to declare a local variable of the same codeunit type, assign it using 'this' (available in newer AL versions) or pass the codeunit as a VAR parameter. In modern AL (BC runtime 11.0+), you can use 'this' keyword to return the current instance. Alternatively, you can declare the return variable and copy state into it.

Correct Pattern:
In AL, use 'this' keyword (BC runtime 11.0+) to return the current codeunit instance: 'exit(this);'. Alternatively, for older runtimes, store state in a SingleInstance codeunit or use a different pattern. The correct fluent API pattern in modern AL is:

procedure SetUrl(Url: Text): Codeunit "CG Request Builder"
begin
    RequestUrl := Url;
    exit(this);
end;
Incorrect Pattern:
exit(Self);

Error Codes: AL0118

6 user-setup-field-reference bc-table-field-names 1 CG-AL-M008

Description: The model generated code that references a field called 'Approval Amount Limit' which does not exist in the current context (likely trying to reference a field from the User Setup table or similar). The model incorrectly assumed the field name or tried to use it without properly accessing the correct table/record. In Business Central, the User Setup table (table 91) has a field called 'Approval Amount' (not 'Approval Amount Limit'), or the model may have been referencing 'Approver ID' and amount limit fields that don't exist with that exact name. The correct field name in the User Setup table is typically 'Purchase Amount Approval Limit' or similar, and the model used an incorrect field name.

Correct Pattern:
Use the correct field name from the User Setup table, such as 'Purchase Amount Approval Limit' (field 8 of table 91 User Setup), or avoid referencing non-existent fields. For dynamic approver determination based on amount thresholds, use UserSetup."Purchase Amount Approval Limit" after getting the User Setup record.
Incorrect Pattern:
"Approval Amount Limit"

Error Codes: AL0118

7 multiline-string-literals al-string-syntax 1 CG-AL-E050

Description: The model attempted to use backtick (`) characters for multiline string literals, which is not valid AL syntax. AL does not support backtick-delimited multiline strings like some other languages (e.g., JavaScript template literals). In AL, multiline text literals were introduced in later runtime versions using a specific syntax (e.g., text constants with line breaks or the newer AL multiline string literal syntax). The model likely confused AL's multiline string syntax with another language's syntax, producing compilation errors starting at line 8 with 'Unexpected character `'. The correct approach in AL would be to either use string concatenation with CR/LF characters, or use the proper AL multiline string literal syntax (available in newer runtimes) which uses single quotes with line continuation, not backticks.

Correct Pattern:
In AL, multiline strings should use the TextConst approach, string concatenation with CR/LF, or (in runtime 12.0+) the proper AL multiline string literal syntax. For example:

procedure GetSqlQuery(): Text
var
    CrLf: Text[2];
begin
    CrLf[1] := 13;
    CrLf[2] := 10;
    exit('SELECT CustomerNo, Name, Balance' + CrLf + 'FROM Customer' + CrLf + 'WHERE Active = true' + CrLf + 'ORDER BY Name');
end;

Alternatively, if the runtime supports it, use the proper AL multiline text literal syntax (not backticks).
Incorrect Pattern:
` (backtick-based multiline string literal attempt - exact code not preserved but errors indicate backtick usage at line 8)

Error Codes: AL0183

8 secrettext-unwrap-scope SecretText-handling 1 CG-AL-H016

Description: The model used SecretText.Unwrap() to extract the plain text value from a SecretText variable. However, the Unwrap() method has scope 'OnPrem' and cannot be used in Extension (cloud/SaaS) development. The correct approach for cloud-compatible extensions is to use format functions or specific overloads that accept SecretText. For example, to get the text value from a SecretText in an extension context, one can use the SecretText in an HttpAuthorizationHeader or use IsolatedStorage.Get with SecretText overloads. For string concatenation, one approach is to use SecretStrSubstNo to build the string while it's still SecretText, then if the result must be returned as Text, the task itself may be inherently OnPrem-scoped. The model should have either recognized the need for target=OnPrem in app.json, or used alternative patterns to avoid Unwrap() calls.

Correct Pattern:
For Extension scope: avoid using .Unwrap() directly. If the task requires extracting plain text from SecretText, the project target must be set to 'OnPrem' in app.json ("target": "OnPrem"), or alternative approaches must be used. For BuildAuthHeader, if OnPrem: Result := 'Bearer ' + ApiKey.Unwrap(); with the app configured as target OnPrem.
Incorrect Pattern:
ApiKey.Unwrap()

Error Codes: AL0296

9 reserved-keyword-as-parameter-name al-syntax-reserved-words 1 CG-AL-M020

Description: The model used 'key' as a parameter name in the procedure signature (likely for ParseConfigSettings or another procedure), but 'key' is a reserved keyword in AL. The model also had syntax errors around line 86 involving Dictionary iteration syntax. The task prompt used 'Key' as a parameter name for ExtractWithDefaults, which the model likely translated directly, but in AL 'key' (case-insensitive) is a reserved keyword and cannot be used as an identifier. The model should have used an alternative name like 'KeyName' or 'SearchKey'. Additionally, the model appears to have incorrect syntax for iterating over Dictionary or JsonObject keys.

Correct Pattern:
procedure ExtractWithDefaults(DataJson: JsonObject; KeyName: Text): Text — use a non-reserved word as the parameter name. Also ensure proper Dictionary/JsonObject iteration syntax using 'foreach KeyValue in Dict.Keys() do' pattern.
Incorrect Pattern:
procedure ExtractWithDefaults(DataJson: JsonObject; key: Text): Text

Error Codes: AL0105

10 string-literal-quoting al-syntax-basics 1 CG-AL-E045

Description: The model generated AL code with improperly terminated string identifiers on line 19 of the generated file. The AL0361 error 'Identifier was not properly terminated. Use the character " to terminate the identifier.' indicates the model likely used incorrect quote characters (e.g., smart/curly quotes instead of straight double quotes) or failed to properly close a quoted identifier in the error message string or field name. This is a fundamental AL syntax issue where the model didn't produce valid AL code.

Correct Pattern:
trigger OnValidate()
begin
    if "Odometer End" < "Odometer Start" then
        if "Odometer Start" <> 0 then
            Error('End reading cannot be lower than start reading.');
    Distance := "Odometer End" - "Odometer Start";
end;
Incorrect Pattern:
// Generated code not found - but line 19 contains an improperly terminated identifier, likely using wrong quote characters in the error message string such as: Error('End reading cannot be lower than start reading.');  or a field reference with mismatched quotes

Error Codes: AL0361

11 interface-and-implementation-syntax interface-definition 1 CG-AL-M009

Description: The model failed to generate valid AL code for the interface and implementation pattern. The compilation errors at line 92 indicate a syntax error ('end' expected, ';' expected, '}' expected), suggesting the model produced malformed AL code - likely incorrect syntax in the interface definition or the implementing codeunit. The generated code file was noted as 'not found' in the analysis but compilation was attempted on CG-AL-M009.al which had syntax errors. The model doesn't properly understand AL interface declaration syntax (which has no object ID, uses 'interface' keyword) and/or the 'implements' keyword on codeunits. Additionally, the test references a mock codeunit 'CG-AL-M009 Mock Shipping' that the model would also need to produce or that should be provided separately - but the primary failure is the model's inability to produce syntactically correct AL interface and implementation code.

Correct Pattern:
interface "Shipping Provider"
{
    procedure CalculateShippingCost(Weight: Decimal; FromCountry: Text; ToCountry: Text): Decimal;
    procedure EstimateDeliveryTime(FromCountry: Text; ToCountry: Text; ServiceType: Text): Integer;
    procedure CreateShipment(OrderNumber: Text; FromAddress: Text; ToAddress: Text; Weight: Decimal): Text[50];
    procedure TrackShipment(TrackingNumber: Text): Text[100];
    procedure ValidateAddress(Street: Text; City: Text; State: Text; ZipCode: Text; Country: Text): Boolean;
}

codeunit 70004 "Standard Shipping Provider" implements "Shipping Provider"
{
    // implement all interface procedures here
}
Incorrect Pattern:
// Generated code not found - but compilation was attempted on CG-AL-M009.al with syntax errors at line 92

Error Codes: AL0104

12 dictionary-key-iteration-syntax dictionary-foreach-syntax 1 CG-AL-H020

Description: The model used 'Key' as a variable name or keyword incorrectly when iterating over Dictionary keys. In AL, 'Key' is a reserved word in certain contexts. The model likely wrote something like 'foreach Key in Dict.Keys do' or used 'Key' as a variable identifier at lines 52 and 95, which caused AL0519 ('Key' is not valid value in this context) and subsequent syntax errors. The correct pattern in AL for iterating dictionary keys is to use a properly declared variable name (not 'Key') and use the Dictionary.Keys() method with a foreach loop, e.g., 'foreach MyKey in Dict.Keys do'.

Correct Pattern:
Declare a variable with a non-reserved name (e.g., 'DictKey: Text;') and iterate using 'foreach DictKey in Dict.Keys do'. Alternatively, use 'foreach DictKey in Dict.Keys() do' depending on the BC version. The identifier 'Key' conflicts with AL reserved usage and should be avoided as a variable name.
Incorrect Pattern:
Key in Dict.Keys

Error Codes: AL0519

13 string-literal-quoting-in-calcformula flowfield-calcformula-syntax 1 CG-AL-M112

Description: The model generated a CalcFormula expression with improperly terminated string identifiers. The AL0361 error 'Identifier was not properly terminated' at line 32 column 81 indicates the model likely used incorrect quoting (e.g., single quotes or unescaped double quotes) within the CalcFormula property for FlowField definitions referencing table/field names like "CG Time Entry" and "Project No.". In AL, identifiers containing spaces in CalcFormula expressions must be enclosed in double quotes, and when inside a string property value, these need to be properly escaped. The model failed to produce syntactically valid CalcFormula expressions for the FlowField fields 'Total Posted Hours' and 'Total Open Hours'.

Correct Pattern:
CalcFormula = sum("CG Time Entry".Hours where("Project No." = field("No."), Posted = const(true)));
Incorrect Pattern:
Not available - generated code was not captured, but compilation errors point to line 32 column 81 with unterminated identifier in CalcFormula

Error Codes: AL0361

14 empty-or-missing-code-generation codeunit-definition 1 CG-AL-H009

Description: The model failed to generate any valid AL code. The generated code was either empty or contained only a comment/placeholder that is not valid AL syntax. The compilation errors ('{' expected, '}' expected, expected application object keyword) at position 1:38 indicate the file content was not a valid AL object definition at all. The task and test are perfectly valid - the model simply failed to produce the required codeunit implementing currency rounding patterns.

Correct Pattern:
codeunit 70213 "CG Price Calculator"
{
    Access = Public;

    var
        Currency: Record Currency;

    procedure CalculateLineAmount(UnitPrice: Decimal; Quantity: Decimal; DiscountPercent: Decimal; CurrencyCode: Code[10]): Decimal
    var
        LineAmount: Decimal;
        Precision: Decimal;
    begin
        LineAmount := UnitPrice * Quantity;
        LineAmount := LineAmount * (1 - DiscountPercent / 100);
        Precision := GetAmountRoundingPrecision(CurrencyCode);
        exit(Round(LineAmount, Precision, '='));
    end;

    procedure CalculateUnitPriceFromAmount(TotalAmount: Decimal; Quantity: Decimal; CurrencyCode: Code[10]): Decimal
    var
        UnitPrice: Decimal;
        Precision: Decimal;
    begin
        if Quantity = 0 then
            exit(0);
        UnitPrice := TotalAmount / Quantity;
        Precision := GetUnitAmountRoundingPrecision(CurrencyCode);
        exit(Round(UnitPrice, Precision, '='));
    end;

    procedure RoundAmount(Amount: Decimal; CurrencyCode: Code[10]; Direction: Text[1]): Decimal
    var
        Precision: Decimal;
    begin
        Precision := GetAmountRoundingPrecision(CurrencyCode);
        exit(Round(Amount, Precision, Direction));
    end;

    procedure GetVATAmount(BaseAmount: Decimal; VATPercent: Decimal; CurrencyCode: Code[10]): Decimal
    var
        VATAmount: Decimal;
        Precision: Decimal;
    begin
        VATAmount := BaseAmount * VATPercent / 100;
        Precision := GetAmountRoundingPrecision(CurrencyCode);
        exit(Round(VATAmount, Precision, '='));
    end;

    local procedure GetAmountRoundingPrecision(CurrencyCode: Code[10]): Decimal
    begin
        if CurrencyCode = '' then
            exit(0.01);
        if Currency.Get(CurrencyCode) then begin
            Currency.InitRoundingPrecision();
            exit(Currency."Amount Rounding Precision");
        end;
        exit(0.01);
    end;

    local procedure GetUnitAmountRoundingPrecision(CurrencyCode: Code[10]): Decimal
    begin
        if CurrencyCode = '' then
            exit(0.00001);
        if Currency.Get(CurrencyCode) then begin
            Currency.InitRoundingPrecision();
            exit(Currency."Unit-Amount Rounding Precision");
        end;
        exit(0.00001);
    end;
}
Incorrect Pattern:
// Generated code not found

Error Codes: AL0104

15 complex-report-and-codeunit-generation report-definition-with-helper-codeunit 1 CG-AL-M007

Description: The model failed to generate valid AL code for a complex task requiring both a Report object (Sales Performance Analysis, ID 70001) and a helper codeunit (CG-AL-M007 Mock Calculator). The compilation error AL0104 at line 327 indicates a syntax error in the generated code (likely a misplaced token or malformed expression). The generated code was not captured/found but the syntax error at line 327 suggests the model produced a large file with a syntax mistake deep in the code, possibly in a complex expression, variable declaration, or trigger body. The task requires generating a report with multiple data items, complex triggers, and a separate mock calculator codeunit with methods like Initialize(), AddSalesLine(), GetRunningTotalByCustomer(), GetRunningTotalByRegion(), CalculateAverageOrderValue(), GetCustomerRank(), GetTopProduct(), GetProductSalesQuantity(), CalculateYoYComparison(), CalculateOrderFrequency(), GetTotalSales(), and GetCustomerCount(). The model likely struggled with the complexity of generating both objects correctly with proper AL syntax.

Correct Pattern:
The code should include: 1) A Report 70001 'Sales Performance Analysis' with Customer, Sales Header, and Sales Line data items, proper triggers, request page, and dataset columns. 2) A Codeunit 'CG-AL-M007 Mock Calculator' with Dictionary-based storage for customer totals, region totals, and product quantities, implementing all required calculation methods with proper AL syntax including semicolons after all statements.
Incorrect Pattern:
// Generated code not available - syntax error at line 327:36 suggests malformed expression or missing semicolon in a complex code block

Error Codes: AL0104

16 rec-xrec-onmodify-trigger trigger-implementation 1 CG-AL-H005

Description: The model failed to generate the correct AL code for this task. The tests show that the OnModify trigger on the 'CG Tracked Item' table is not creating audit log entries when fields change. The generated code was not found/captured, but the test failures indicate that either: (1) the OnModify trigger was not implemented at all, (2) the trigger was implemented incorrectly and doesn't properly compare Rec vs xRec to detect changes, or (3) the audit log insertion logic is missing or broken. The key AL concept here is using xRec in the OnModify trigger to detect field-level changes and creating corresponding audit log records. The model needed to implement an OnModify trigger that compares Rec."Unit Price" <> xRec."Unit Price" and (not xRec.Blocked) and Rec.Blocked to create appropriate audit log entries.

Correct Pattern:
trigger OnModify()
var
    AuditLog: Record "CG Audit Log";
begin
    if Rec."Unit Price" <> xRec."Unit Price" then begin
        AuditLog.Init();
        AuditLog."Table ID" := Database::"CG Tracked Item";
        AuditLog."Record ID" := Rec.RecordId;
        AuditLog."Field Changed" := 'Unit Price';
        AuditLog."Old Value" := Format(xRec."Unit Price");
        AuditLog."New Value" := Format(Rec."Unit Price");
        AuditLog."Changed At" := CurrentDateTime;
        AuditLog."Changed By" := CopyStr(UserId, 1, 50);
        AuditLog.Insert(true);
    end;
    if (not xRec.Blocked) and Rec.Blocked then begin
        AuditLog.Init();
        AuditLog."Table ID" := Database::"CG Tracked Item";
        AuditLog."Record ID" := Rec.RecordId;
        AuditLog."Field Changed" := 'Blocked';
        AuditLog."Old Value" := 'No';
        AuditLog."New Value" := 'Yes';
        AuditLog."Changed At" := CurrentDateTime;
        AuditLog."Changed By" := CopyStr(UserId, 1, 50);
        AuditLog.Insert(true);
    end;
end;
Incorrect Pattern:
// Generated code not found - model likely produced incomplete or incorrect OnModify trigger implementation

Error Codes: Assert.IsFalse failed. Audit log entry should exist for price change

17 parse-failure unknown 1 CG-AL-E052

Description: Failed to parse LLM analysis response: Looking at the failure details: 1. The generated code compiled successfully and was published 2. Most tests pass, but `TestDateToText_ValidDate` and `TestDateToText_FirstDayOfYear` fail 3. The test f

Correct Pattern:
Incorrect Pattern: