← Back to Benchmark Results

anthropic/claude-sonnet-4-5-20250929

66.1%
Pass Rate
37/56
Tasks Passed
3
Runs
63.1%
pass@1
66.1%
pass@3
94.6%
Consistency
0.1
Temperature
-
Thinking
454,963
Tokens
$4.27
Cost
1st: 852nd: 21Failed: 1937/56 passed

Known Shortcomings (8)

Sorted by occurrence count (most frequent first)

# Concept AL Concept Count Affected Tasks
1 multiline-string-literals text-literal-syntax 1 CG-AL-E050

Description: The model attempted to use raw multiline string literals by embedding newlines directly within single-quoted strings. AL does not support multiline string literals in this way — a text literal must be terminated on the same line. The task description says 'Use a multiline string literal, not string concatenation', which likely refers to AL's newer multiline text literal syntax (available in newer AL versions) or requires a specific approach. In classic AL, multiline strings must be constructed using concatenation with CR/LF characters or using the newer AL multiline string literal syntax (triple single quotes or similar). The model incorrectly assumed that simply breaking a single-quoted string across multiple lines would work, which causes AL0360 'Text literal was not properly terminated' errors.

Correct Pattern:
In AL, multiline text can be built using concatenation with newline characters, e.g.:

var
    CrLf: Text[2];
begin
    CrLf[1] := 13;
    CrLf[2] := 10;
    SqlQuery := 'SELECT CustomerNo, Name, Balance' + CrLf + 'FROM Customer' + CrLf + 'WHERE Active = true' + CrLf + 'ORDER BY Name';

Note: Despite the task saying 'use a multiline string literal, not string concatenation', AL does not support raw multiline string literals in the way the model attempted. The task description may be misleading, but the model should know AL's actual string literal rules.
Incorrect Pattern:
SqlQuery := 'SELECT CustomerNo, Name, Balance
FROM Customer
WHERE Active = true
ORDER BY Name';

Error Codes: AL0360

2 query-filter-element-syntax query-definition 1 CG-AL-H011

Description: The model incorrectly specified the ColumnFilter property on a filter element. In AL query objects, when using a 'filter' element, the ColumnFilter should reference the filter element's name (not the source field name with quotes). The correct syntax is 'ColumnFilter = Document_Type = const(Order)' rather than 'ColumnFilter = "Document Type" = const(Order)'. The filter element name 'Document_Type' is the identifier used in the ColumnFilter, not the source field name 'Document Type'. The compiler error AL0186 indicates that the reference '"Document Type"' does not exist in the context of the application object 'Document_Type'.

Correct Pattern:
filter(Document_Type; "Document Type")
{
    ColumnFilter = Document_Type = const(Order);
}
Incorrect Pattern:
filter(Document_Type; "Document Type")
{
    ColumnFilter = "Document Type" = const(Order);
}

Error Codes: AL0186

3 jsonobject-get-method-signature json-api-usage 1 CG-AL-H014

Description: The model incorrectly used JsonObject.Get('key') as a single-parameter function that returns a JsonToken directly. In AL, JsonObject.Get() requires two parameters: Get(Key: Text, var Result: JsonToken) and returns Boolean. The model also incorrectly called JsonValue.Get() which does not exist — JsonValue has AsText(), AsInteger(), AsDecimal(), AsBoolean() methods instead. The model confused the AL JSON API with other language JSON APIs.

Correct Pattern:
var NameToken: JsonToken;
if CustomerJson.Get('name', NameToken) then
    Name := NameToken.AsValue().AsText();
if CustomerJson.Get('age', AgeToken) then
    Age := AgeToken.AsValue().AsInteger();
if CustomerJson.Get('active', ActiveToken) then
    Active := ActiveToken.AsValue().AsBoolean();
// For decimals: Amount := AmountToken.AsValue().AsDecimal();
Incorrect Pattern:
CustomerJson.Get('name').AsValue().Get(Name);
ItemObject.Get('quantity').AsValue().Get(Quantity)
ResultToken.AsValue().Get(ResultValue)
AmountToken.AsValue().Get(Amount)

Error Codes: AL0135

4 cross-join-dataitem-link-constraints query-dataitem-linking 1 CG-AL-H017

Description: The model incorrectly specified a DataItemLink on the 'Dimension_Set_Entry' dataitem that references a field from the 'Department' dataitem (the grandparent), but in AL queries, a Column or Filter source must reference a field defined on the table of its parent DataItem. The DataItemLink on 'Dimension_Set_Entry' uses 'Department."Dimension Code"' and 'Department.Code', but the parent DataItem of 'Dimension_Set_Entry' is 'Project' (which references 'Dimension Value'). Additionally, with CrossJoin between Department and Project, the linking semantics are complex. The model should have either linked to the parent 'Project' dataitem's fields or used a different linking strategy that is valid for the LeftOuterJoin on 'Dimension Set Entry'. For a LeftOuterJoin, the DataItemLink must reference fields from the immediate parent dataitem's table.

Correct Pattern:
The DataItemLink on 'Dimension_Set_Entry' must only reference fields from its immediate parent dataitem 'Project'. For example:
DataItemLink = "Dimension Code" = Project."Dimension Code", "Dimension Value Code" = Project.Code;
Alternatively, if the intent is to link to both Department and Project, the query structure needs to be redesigned since AL query DataItemLink can only reference the parent dataitem's fields.
Incorrect Pattern:
dataitem(Dimension_Set_Entry; "Dimension Set Entry")
{
    SqlJoinType = LeftOuterJoin;
    DataItemLink = "Dimension Code" = Department."Dimension Code",
                   "Dimension Value Code" = Department.Code;
    column(MatchCount; "Entry No.")
    {
        Method = Count;
    }
}

Error Codes: AL0345

5 reserved-keyword-as-variable-name al-reserved-words 1 CG-AL-H020

Description: The model used 'Key' as a variable name in the MergeDictionaries and GetKeys procedures. In AL, 'Key' is a reserved word (it's used in table definitions for key groups), so it cannot be used as a variable identifier. The compiler raises AL0519 ('Key' is not valid value in this context) because it interprets 'Key' as a keyword rather than a variable name. The model should have used a different variable name like 'DictKey' or 'KeyValue'.

Correct Pattern:
var
    DictKey: Text;
...
foreach DictKey in Keys do
Incorrect Pattern:
var
    Key: Text;
...
foreach Key in Keys do

Error Codes: AL0519

6 reserved-keyword-as-variable-name al-reserved-keywords 1 CG-AL-M021

Description: The model used 'Key' as a variable name in AL, but 'Key' is a reserved keyword in AL. This causes compilation errors AL0519 and AL0105. The variable declarations on line 84 ('Key: Text;') and usage on line 139 ('Key: Text') fail because 'Key' cannot be used as an identifier. The model should have used a different variable name like 'KeyName', 'YamlKey', or 'PropertyKey' to avoid the reserved keyword conflict.

Correct Pattern:
YamlKey: Text;
...
Value := CopyStr(Line, ColonPos + 1).Trim();
...
local procedure GetJsonValue(JsonObj: JsonObject; KeyName: Text): Text
Incorrect Pattern:
Key: Text;
...
Value := CopyStr(Line, ColonPos + 1).Trim();
...
local procedure GetJsonValue(JsonObj: JsonObject; Key: Text): Text

Error Codes: AL0519

7 http-headers-get-pattern http-client-api 1 CG-AL-M005

Description: The model incorrectly called GetHeaders() as a function returning a value (fluent style), but in AL, HttpContent.GetHeaders() requires a var HttpHeaders parameter. The correct pattern is to declare an HttpHeaders variable, call GetHeaders(Headers) to populate it, then use Headers.Clear() and Headers.Add(). The same error occurs on RequestMessage.GetHeaders(). This is a common AL HTTP API pattern the model failed to use correctly.

Correct Pattern:
var
    ContentHeaders: HttpHeaders;
    RequestHeaders: HttpHeaders;
begin
    Content.GetHeaders(ContentHeaders);
    ContentHeaders.Clear();
    ContentHeaders.Add('Content-Type', 'application/json');
    ...
    RequestMessage.GetHeaders(RequestHeaders);
    RequestHeaders.Add('Authorization', AuthTokenTxt);
Incorrect Pattern:
Content.GetHeaders().Clear();
Content.GetHeaders().Add('Content-Type', 'application/json');
...
RequestMessage.GetHeaders().Add('Authorization', AuthTokenTxt);

Error Codes: AL0135

8 report-rendering-layout-syntax report-definition 1 CG-AL-M007

Description: The model generated a report with a 'rendering' section that references an RDLC layout file ('SalesPerformanceAnalysis.rdl') that doesn't exist. More critically, the compilation errors at lines 140 and 168 indicate structural syntax issues in the report definition. The AL0104 errors ('Syntax error, } expected') at lines 140 and 168 point to problems with the 'rendering' block syntax. In AL for Business Central, the 'rendering' section with 'layout' keyword may not be supported in the version being compiled, or the syntax used is incorrect. The model should have either used the older 'RDLCLayout' property at the report level, or ensured the rendering block syntax matches the compiler version. Additionally, the referenced .rdl layout file doesn't exist, which would cause further issues. The AL0198 error at line 202 is a cascading error from the earlier syntax failures, where the compiler sees the codeunit definition as unexpected because it couldn't properly parse the report object.

Correct Pattern:
Either use the report-level property 'RDLCLayout = './SalesPerformanceAnalysis.rdl';' (and provide the file), or use 'DefaultLayout = RDLC;' with a proper rendering section compatible with the target BC version. For compilation without an actual layout file, remove the rendering section and the DefaultRenderingLayout property entirely, using 'ProcessingOnly = true;' if no layout is needed.
Incorrect Pattern:
    rendering
    {
        layout(SalesPerformanceLayout)
        {
            Type = RDLC;
            LayoutFile = 'SalesPerformanceAnalysis.rdl';
        }
    }

Error Codes: AL0104