← Back to Benchmark Results

openrouter/deepseek/deepseek-v3.2

55.4%
Pass Rate
31/56
Tasks Passed
3
Runs
47.6%
pass@1
55.4%
pass@3
80.4%
Consistency
0.1
Temperature
-
Thinking
441,107
Tokens
$3.65
Cost
1st: 552nd: 25Failed: 2531/56 passed

Known Shortcomings (18)

Sorted by occurrence count (most frequent first)

# Concept AL Concept Count Affected Tasks
1 dictionary-clear-method al-collection-types 1 CG-AL-E032

Description: The model generated a mock codeunit implementing the 'CG Token Provider' interface but used '.Clear' on Dictionary variables, which is not a valid method in AL. In AL, the correct method to remove all entries from a Dictionary is '.Values.RemoveAll()' or simply reinitializing the variable via 'Clear(VariableName)' using the built-in Clear procedure. The model confused the dot-notation method call with the AL built-in Clear() function. The compilation errors AL0132 indicate that 'Dictionary of [Text, Text]' and 'Dictionary of [Text, DateTime]' do not have a 'Clear' member method.

Correct Pattern:
Clear(SomeDictionary);
Clear(SomeOtherDictionary);
Incorrect Pattern:
SomeDictionary.Clear;
SomeOtherDictionary.Clear;

Error Codes: AL0132

2 application-area-in-page-extension-field page-extension-properties 1 CG-AL-E006

Description: The model generated a page extension with 'ApplicationArea' property set directly on fields within a 'addlast' or 'addfirst' group modification. In newer versions of Business Central AL, the ApplicationArea property cannot be applied at the individual field level within certain page extension contexts (it may need to be set at the page level or omitted when the app's app.json specifies application areas). The AL0124 error 'The property ApplicationArea cannot be used in this context' indicates the model placed ApplicationArea on page extension fields where it is not allowed. The model should have omitted the ApplicationArea property from the field controls in the page extension, or used it only where permitted.

Correct Pattern:
field("Preferred Contact Method"; Rec."Preferred Contact Method")
            {
                Caption = 'Preferred Contact Method';
            }

// ApplicationArea should not be specified on fields in page extensions when the runtime version / app.json configuration handles it globally, or it should be placed at the correct scope.
Incorrect Pattern:
field("Preferred Contact Method"; Rec."Preferred Contact Method")
            {
                ApplicationArea = All;
                Caption = 'Preferred Contact Method';
            }

Error Codes: AL0124

3 multiline-string-literals text-literal-syntax 1 CG-AL-E050

Description: The model attempted to use actual multiline string literals (spanning multiple lines in source code) which is not valid AL syntax in most BC versions. The AL0360 errors ('Text literal was not properly terminated') indicate the model broke a string across multiple lines without proper termination. The model should have either used string concatenation with CR/LF characters, or used the proper AL multiline text literal syntax if available in the target runtime. The task explicitly asks for multiline string literals, but the model didn't know the correct AL syntax to produce multiline text content - it likely tried to embed raw newlines inside single-quoted strings, which the AL compiler rejects.

Correct Pattern:
In AL, multiline text content should be built using concatenation with CR/LF characters, e.g.:
var CrLf: Text[2];
begin
  CrLf[1] := 13;
  CrLf[2] := 10;
  exit('SELECT CustomerNo, Name, Balance' + CrLf + 'FROM Customer' + CrLf + 'WHERE Active = true' + CrLf + 'ORDER BY Name');
end;

Alternatively, in newer AL runtimes (14.0+), use the proper multiline text literal syntax if supported.
Incorrect Pattern:
exit('SELECT CustomerNo, Name, Balance
FROM Customer
WHERE Active = true
ORDER BY Name');

Error Codes: AL0360

4 page-extension-cardpageid-override page-extension-syntax 1 CG-AL-E053

Description: The model failed to generate valid AL code for a page extension. The compilation errors at line 3 (syntax error, ';' expected) and line 24 suggest the model produced syntactically invalid AL code, likely struggling with the BC 2025 Wave 1 feature of overriding CardPageId on a list page extension. The model either didn't know the correct syntax for the CardPageId property override in a page extension context, or produced malformed code with incorrect structure. The 'Generated code not found' note and the multiple syntax errors at lines 3 and 24 indicate the model wrote something fundamentally wrong - possibly using incorrect keywords or property placement for the new CardPageId override feature.

Correct Pattern:
pageextension 70053 "CG Item List Extension" extends "Item List"
{
    CardPageId = "Item Card";

    actions
    {
        addlast(Processing)
        {
            action("Show Item Details")
            {
                ApplicationArea = All;
                Caption = 'Show Item Details';
                Image = Card;
                RunObject = page "Item Card";
                RunPageLink = "No." = field("No.");
            }
        }
    }
}
Incorrect Pattern:
// Generated code not available but errors at line 3:22 and 24:25 suggest malformed pageextension declaration and action definition

Error Codes: AL0104

5 errorinfo-custom-dimensions-api ErrorInfo-API 1 CG-AL-H007

Description: The model used 'AddCustomDimension' and 'GetCustomDimension' methods on ErrorInfo, which do not exist. In AL, ErrorInfo's CustomDimensions is a Dictionary of [Text, Text] that you interact with using the Dictionary methods. To add custom dimensions, you must get the CustomDimensions dictionary and use Dictionary.Add() or Dictionary.Set(). To read them, you use Dictionary.Get(). The model did not know the correct API surface for ErrorInfo.CustomDimensions in AL.

Correct Pattern:
var Dims: Dictionary of [Text, Text]; ... Dims.Add('ErrorCode', Format(ErrorCode.AsInteger())); Dims.Add('FieldName', FieldName); ErrInfo.CustomDimensions(Dims); // To read: ErrInfo.CustomDimensions().Get('ErrorCode', ResultText);
Incorrect Pattern:
ErrInfo.AddCustomDimension('ErrorCode', ...); ErrInfo.GetCustomDimension('ErrorCode');

Error Codes: AL0132

6 codeunit-generation-empty-output basic-codeunit-definition 1 CG-AL-H014

Description: The model failed to generate any AL code at all. The compilation error AL0198 'Expected one of the application object keywords' indicates the generated file was empty or contained no valid AL object declaration. The model should have produced a codeunit 70014 'CG JSON Parser' with the four specified procedures (ParseCustomerData, ProcessOrderItems, SafeGetText, ExtractNestedValue) using JsonObject/JsonArray/JsonToken/JsonValue types. The task and test are both valid - the model simply did not produce any output.

Correct Pattern:
codeunit 70014 "CG JSON Parser"
{
    procedure ParseCustomerData(CustomerJson: JsonObject): Text
    var
        NameToken: JsonToken;
        AgeToken: JsonToken;
        ActiveToken: JsonToken;
        NameVal: Text;
        AgeVal: Integer;
        ActiveVal: Boolean;
    begin
        CustomerJson.Get('name', NameToken);
        NameVal := NameToken.AsValue().AsText();
        CustomerJson.Get('age', AgeToken);
        AgeVal := AgeToken.AsValue().AsInteger();
        CustomerJson.Get('active', ActiveToken);
        ActiveVal := ActiveToken.AsValue().AsBoolean();
        exit(StrSubstNo('Name: %1, Age: %2, Active: %3', NameVal, AgeVal, ActiveVal));
    end;

    procedure ProcessOrderItems(OrderJson: JsonObject): Integer
    var
        ItemsToken: JsonToken;
        ItemsArray: JsonArray;
        ItemToken: JsonToken;
        ItemObj: JsonObject;
        QtyToken: JsonToken;
        Total: Integer;
        i: Integer;
    begin
        OrderJson.Get('items', ItemsToken);
        ItemsArray := ItemsToken.AsArray();
        for i := 0 to ItemsArray.Count() - 1 do begin
            ItemsArray.Get(i, ItemToken);
            ItemObj := ItemToken.AsObject();
            ItemObj.Get('quantity', QtyToken);
            Total += QtyToken.AsValue().AsInteger();
        end;
        exit(Total);
    end;

    procedure SafeGetText(Obj: JsonObject; Key: Text; DefaultValue: Text): Text
    var
        Token: JsonToken;
    begin
        if Obj.Get(Key, Token) then
            exit(Token.AsValue().AsText());
        exit(DefaultValue);
    end;

    procedure ExtractNestedValue(RootJson: JsonObject): Decimal
    var
        Token: JsonToken;
        DataObj: JsonObject;
        DetailsObj: JsonObject;
    begin
        RootJson.Get('data', Token);
        DataObj := Token.AsObject();
        DataObj.Get('details', Token);
        DetailsObj := Token.AsObject();
        DetailsObj.Get('amount', Token);
        exit(Token.AsValue().AsDecimal());
    end;
}
Incorrect Pattern:

Error Codes: AL0198

7 dimension-table-api query-object-definition 1 CG-AL-H017

Description: The model generated code that references a non-existent method 'CodeExists' on the 'Dimension' record. The compilation errors (AL0132) indicate the model tried to call 'Dimension.CodeExists' which doesn't exist in the BC API. The model likely attempted to add validation logic or helper code that isn't part of the standard Dimension table API, rather than simply defining the query object with the correct dataitems, columns, filters, and join types as specified in the task. The model should have produced a straightforward query object with CrossJoin dataitems on 'Dimension Value' tables with appropriate filters and columns, plus a LeftOuterJoin to 'Dimension Set Entry' with a Count column.

Correct Pattern:
query 70017 "CG Dimension Matrix"
{
    QueryType = Normal;

    elements
    {
        dataitem(DeptDimValue; "Dimension Value")
        {
            SqlJoinType = CrossJoin;
            DataItemTableFilter = "Dimension Code" = const('DEPARTMENT');
            column(DepartmentCode; "Code") { }
            column(DepartmentName; Name) { }

            dataitem(ProjDimValue; "Dimension Value")
            {
                SqlJoinType = CrossJoin;
                DataItemTableFilter = "Dimension Code" = const('PROJECT');
                column(ProjectCode; "Code") { }
                column(ProjectName; Name) { }

                dataitem(DimSetEntry; "Dimension Set Entry")
                {
                    SqlJoinType = LeftOuterJoin;
                    DataItemLink = ... ;
                    column(MatchCount; "Dimension Value ID")
                    {
                        Method = Count;
                    }
                }
            }
        }
    }
}
Incorrect Pattern:
Record Dimension does not contain a definition for 'CodeExists' (referenced at lines 58 and 60)

Error Codes: AL0132

8 fluent-api-return-self-pattern codeunit-self-reference 1 CG-AL-H018

Description: The model attempted to use 'CurrCodeunit' to return a reference to the current codeunit instance for fluent API chaining, but 'CurrCodeunit' does not exist in AL. In AL, codeunits are value types and there is no built-in keyword to reference the current codeunit instance. The correct pattern for a fluent API in AL is to declare the return type as the codeunit type and use a local variable or the 'this' implicit reference via returning the codeunit variable. The typical approach is to use a 'var' parameter pattern or to declare a local variable of the codeunit type, copy state, and return it — but more practically, the procedures should accept and return the codeunit by 'var' parameter, or use the pattern where each method returns a Codeunit variable that was passed in. Additionally, the model used 'Dictionary' as a type name which is not valid in AL — the correct type is 'Dictionary of [Text, Text]'.

Correct Pattern:
In AL, to implement a fluent API pattern, each procedure should be declared as returning 'Codeunit "CG Request Builder"' and use 'exit(this);' or declare a global variable 'Self: Codeunit "CG Request Builder";' initialized in each method. The most common working pattern in modern AL (BC v20+) is: procedure SetUrl(Url: Text): Codeunit "CG Request Builder" begin RequestUrl := Url; exit(this); end; — where 'this' is the implicit self-reference available in newer AL runtimes. For the Dictionary type, use 'Dictionary of [Text, Text]' not just 'Dictionary'.
Incorrect Pattern:
exit(CurrCodeunit);

Error Codes: AL0118

9 dictionary-foreach-variable-declaration dictionary-iteration 1 CG-AL-H020

Description: The model generated code that uses a variable named 'Key' in dictionary iteration (foreach loops or similar constructs) without declaring it as a local variable first. In AL, all variables used in a procedure must be declared in the 'var' section. The error AL0118 'The name Key does not exist in the current context' appears multiple times at lines associated with MergeDictionaries and GetKeys procedures, indicating the model forgot to declare the loop variable 'Key' in the local var section of those procedures. This is a common AL knowledge gap where the model doesn't properly declare all local variables needed for dictionary key iteration.

Correct Pattern:
var
    Key: Text;
begin
    foreach Key in Dict1.Keys() do begin
        ...
    end;
end;
Incorrect Pattern:
foreach Key in Dict1.Keys() do begin ... end;

Error Codes: AL0118

10 tryfunction-and-runtime-version-awareness RecordRef-dynamic-operations 1 CG-AL-H022

Description: The model generated code with multiple issues: (1) It used TryFunction or similar pattern incorrectly, resulting in 'Operator not cannot be applied to an operand of type None' errors at lines 12, 27, and 101 - likely writing 'if not RecRef.Open(TableId) then' when RecRef.Open() is a void procedure that doesn't return a Boolean. The correct pattern is to use a [TryFunction] helper or wrap in a try pattern. (2) It used RecordRef.FieldExist() and RecordRef.Field() which are only available in runtime 16.0+, but the target environment uses runtime 15.0. The model should have used FieldRef with error handling instead, or checked field existence via a TryFunction pattern. These are fundamental AL/BC runtime version awareness and RecordRef API knowledge gaps.

Correct Pattern:
RecRef.Open() is void - use a [TryFunction] procedure to handle invalid table IDs. For runtime 15.0, use FieldRef := RecRef.FieldIndex() iteration or TryFunction wrappers instead of FieldExist()/Field() which require runtime 16.0+.
Incorrect Pattern:
if not RecRef.Open(TableId) then ... RecRef.FieldExist() ... RecRef.Field()

Error Codes: AL0173

11 al-syntax-expression-error interface-implementation 1 CG-AL-M009

Description: The model generated code with a syntax error at line 232, likely involving an incorrect expression or string interpolation pattern. The errors (AL0224 'Expression expected', AL0104 'Syntax error') at column 39-42 suggest the model used an invalid syntax construct, possibly C#-style string interpolation (e.g., $'...' or StrSubstNo with incorrect formatting), an invalid type cast, or some other non-AL syntax pattern. The generated code was not captured but the compilation errors clearly indicate the model produced syntactically invalid AL code in the implementation codeunit. This is a model knowledge gap - the task and tests are valid, but the model doesn't properly know AL expression syntax.

Correct Pattern:
Use proper AL expression syntax. For string operations use StrSubstNo('%1-%2', Var1, Var2) or simple concatenation with '+'. For type conversions use Format() function. Avoid C#-style interpolation or other non-AL patterns.
Incorrect Pattern:
Line 232 (not captured, but contains syntax error around column 39-42)

Error Codes: AL0224

12 reserved-keyword-as-parameter-name al-reserved-keywords 1 CG-AL-M020

Description: The model used 'key' as a parameter name in the ExtractWithDefaults procedure, but 'key' is a reserved keyword in AL. The task definition specifies the parameter name as 'Key', which is also a reserved keyword. The model should have used a different parameter name (e.g., 'KeyName' or 'SearchKey') or escaped it to avoid the AL0105 syntax error. The task definition uses 'Key' as the parameter name which is problematic, but the model should know to avoid reserved keywords in AL.

Correct Pattern:
procedure ExtractWithDefaults(DataJson: JsonObject; KeyName: Text): Text
Incorrect Pattern:
procedure ExtractWithDefaults(DataJson: JsonObject; key: Text): Text

Error Codes: AL0105

13 reserved-keyword-as-variable-name al-syntax-reserved-words 1 CG-AL-M021

Description: The model used 'Key' as a variable name in AL code, but 'Key' is a reserved keyword in AL (used for key definitions in table objects). This caused AL0519 errors. Additionally, the model used '#' characters in string expressions (likely trying to use char codes like '#10' for newline), which the AL compiler interprets as preprocessor directives, causing AL0620 errors. The model lacks knowledge of AL reserved words and proper string/character handling in AL.

Correct Pattern:
Use a non-reserved variable name like 'KeyName' or 'PropertyKey' instead of 'Key'. For newline characters, use a Text variable with character assignment like Lf[1] := 10 rather than inline '#' notation.
Incorrect Pattern:
Key (at lines 90 and 127) used as variable/identifier; '#' used inline in expressions at lines 134, 147, 215

Error Codes: AL0519

14 string-numeric-extraction-and-increment text-parsing-and-manipulation 1 CG-AL-E051

Description: The model generated code that incorrectly handles extracting and incrementing the numeric portion of a string. The test shows that IncrementByStep('DOC-001', 1) returned 'DOC-000' instead of 'DOC-002', indicating the model's implementation either failed to correctly parse the numeric suffix, failed to increment it properly, or failed to reconstruct the string with the incremented value and proper zero-padding. The task and tests are valid - the model simply wrote incorrect logic for parsing the trailing numeric portion from a text string, incrementing it by the step value, and formatting it back with the original width (leading zeros preserved). This is a string manipulation / text parsing knowledge gap.

Correct Pattern:
procedure IncrementByStep(Value: Text; Step: Integer): Text
var
    i: Integer;
    NumStart: Integer;
    NumStr: Text;
    NumVal: Integer;
    NewNumStr: Text;
    Prefix: Text;
begin
    NumStart := 0;
    for i := StrLen(Value) downto 1 do
        if (Value[i] >= '0') and (Value[i] <= '9') then
            NumStart := i
        else
            break;
    if NumStart = 0 then
        exit(Value);
    Prefix := CopyStr(Value, 1, NumStart - 1);
    NumStr := CopyStr(Value, NumStart);
    Evaluate(NumVal, NumStr);
    NumVal += Step;
    NewNumStr := Format(NumVal);
    while StrLen(NewNumStr) < StrLen(NumStr) do
        NewNumStr := '0' + NewNumStr;
    exit(Prefix + NewNumStr);
end;
Incorrect Pattern:
// Generated code not found - but based on test output, the implementation returns 'DOC-000' for IncrementByStep('DOC-001', 1), suggesting the numeric extraction or increment logic is broken
15 temporary-table-parameter-handling temporary-table 1 CG-AL-H003

Description: The test TestHighInventoryDiscount fails because the generated code does not correctly populate the TempResult temporary table with items that have inventory >= 100. The test finds an Item with Inventory >= 100 and Unit Price > 0, then calls ProcessItemsWithDiscount with MinDiscount=15, and expects to find that item in TempResult with Discount Percent = 15. The failure 'Assert.IsTrue failed. High inventory item should be in results' indicates the item was not found in TempResult after processing. This suggests the model's generated code has a bug in how it populates the temporary table - likely an issue with line numbering, filtering, or the temporary table record insertion logic. Since the generated code was not captured but the app compiled and other tests may have issues, the core problem is the model failed to correctly implement the temporary table processing logic, particularly ensuring items with high inventory are properly added to the result set.

Correct Pattern:
procedure ProcessItemsWithDiscount(var TempResult: Record "CG Discount Result" temporary; MinDiscount: Decimal)
var
    Item: Record Item;
    LineNo: Integer;
    DiscountPct: Decimal;
begin
    TempResult.Reset();
    TempResult.DeleteAll();
    LineNo := 0;

    Item.SetFilter("Unit Price", '>0');
    if Item.FindSet() then
        repeat
            if Item.Inventory >= 100 then
                DiscountPct := 15
            else if Item.Inventory >= 50 then
                DiscountPct := 10
            else if Item.Inventory >= 10 then
                DiscountPct := 5
            else
                DiscountPct := 0;

            if DiscountPct >= MinDiscount then begin
                LineNo += 1;
                TempResult.Init();
                TempResult."Line No." := LineNo;
                TempResult."Item No." := Item."No.";
                TempResult."Original Price" := Item."Unit Price";
                TempResult."Discount Percent" := DiscountPct;
                TempResult."Final Price" := Round(Item."Unit Price" * (1 - DiscountPct / 100), 0.01);
                TempResult.Insert();
            end;
        until Item.Next() = 0;
end;
Incorrect Pattern:
// Generated code not found - but the compiled app failed TestHighInventoryDiscount indicating the item with inventory >= 100 was not found in TempResult after calling ProcessItemsWithDiscount(TempResult, 15)
16 multiline-string-literals al-string-syntax 1 CG-AL-E050

Description: The model failed to generate valid AL code. The compilation errors indicate the model likely used backtick characters (```) for multiline strings, which is not valid AL syntax. AL multiline string literals use a specific syntax (introduced in newer AL versions) or require string concatenation with carriage return/line feed characters. The model either produced no recognizable AL code or used incorrect syntax like backtick-delimited strings (possibly confusing AL with other languages like JavaScript or C#). The AL0198 error at line 42 suggests the codeunit definition ended prematurely or was malformed, and the AL0183 errors about unexpected backtick characters confirm the model tried to use backticks for multiline strings.

Correct Pattern:
In AL, multiline text literals can be constructed using string concatenation with CR/LF characters, e.g.:

procedure GetSqlQuery(): Text
var
    CrLf: Text[2];
begin
    CrLf[1] := 13;
    CrLf[2] := 10;
    exit('SELECT CustomerNo, Name, Balance' + CrLf + 'FROM Customer' + CrLf + 'WHERE Active = true' + CrLf + 'ORDER BY Name');
end;

Alternatively, in AL runtime 12.0+ (BC 2023 wave 2+), verbatim string literals can be used with the @'' syntax or multiline text constants.
Incorrect Pattern:
Code contained backtick characters (`) likely attempting JavaScript/markdown-style multiline strings

Error Codes: AL0198

17 avoid-undefined-table-references-and-duplicate-methods codeunit-implementation-patterns 1 CG-AL-M005

Description: The model generated code that references a table 'Payment Transaction Log' which doesn't exist in the project or BC base app, and also defined a duplicate method 'ExecuteHttpRequest' with the same parameter types. The task asked for a codeunit with specific procedures, and the model should have implemented LogPaymentTransaction without relying on a custom table (e.g., using Message or a simple log mechanism), and should have avoided defining duplicate method signatures. The model failed to understand that: (1) it cannot reference tables that don't exist unless it also creates them, and (2) AL does not allow overloaded methods with identical parameter signatures.

Correct Pattern:
LogPaymentTransaction should use an in-memory approach (e.g., Message, or simply do nothing for audit logging in a test context) rather than referencing a non-existent table. HTTP request helper methods should have unique signatures or be consolidated into a single method. For example:

procedure LogPaymentTransaction(TransactionId: Text[50]; Amount: Decimal; Status: Text[20])
begin
    // Log to message or simply track in-memory without referencing undefined tables
end;

And ensure ExecuteHttpRequest is defined only once with a single signature.
Incorrect Pattern:
Table 'Payment Transaction Log' referenced in LogPaymentTransaction procedure; duplicate ExecuteHttpRequest method definition

Error Codes: AL0185

18 complex-report-with-helper-codeunit report-definition-and-codeunit-structure 1 CG-AL-M007

Description: The model was asked to create a complex report 'Sales Performance Analysis' (ID 70001) along with a supporting codeunit 'CG-AL-M007 Mock Calculator' that the tests depend on. The generated code has a syntax error at line 500 (missing closing brace) and a structural error at line 528 (unexpected object keyword), indicating the model failed to properly structure the AL file containing both the report object and the mock calculator codeunit. The model likely attempted to put both objects in a single file or had mismatched braces in a complex nested structure. The task requires generating a report with multiple data items (Customer, Sales Header, Sales Line) with triggers, plus a separate codeunit implementing methods like Initialize(), AddSalesLine(), GetRunningTotalByCustomer(), GetRunningTotalByRegion(), CalculateAverageOrderValue(), GetCustomerRank(), GetTopProduct(), GetProductSalesQuantity(), CalculateYoYComparison(), CalculateOrderFrequency(), GetTotalSales(), and GetCustomerCount(). The model produced syntactically invalid AL code with mismatched braces and improper object boundaries.

Correct Pattern:
The solution requires two properly structured AL objects: 1) report 70001 'Sales Performance Analysis' with Customer, Sales Header, and Sales Line data items including proper triggers and request page, and 2) a separate codeunit 'CG-AL-M007 Mock Calculator' with Dictionary-based internal storage implementing all required calculation methods (Initialize, AddSalesLine, GetRunningTotalByCustomer, GetRunningTotalByRegion, CalculateAverageOrderValue, GetCustomerRank, GetTopProduct, GetProductSalesQuantity, CalculateYoYComparison, CalculateOrderFrequency, GetTotalSales, GetCustomerCount). Each object must have properly matched braces and be clearly separated.
Incorrect Pattern:
// Generated code around line 500 has mismatched braces causing AL0104, and around line 528 has an object declaration in an unexpected position causing AL0198

Error Codes: AL0104