← Back to Benchmark Results

gemini/gemini-3-pro-preview

64.3%
Pass Rate
36/56
Tasks Passed
3
Runs
55.4%
pass@1
64.3%
pass@3
83.9%
Consistency
0.1
Temperature
-
Thinking
2,469,073
Tokens
$3.03
Cost
1st: 862nd: 7Failed: 2036/56 passed

Known Shortcomings (9)

Sorted by occurrence count (most frequent first)

# Concept AL Concept Count Affected Tasks
1 multiline-string-literals al-string-syntax 1 CG-AL-E050

Description: The model attempted to use backtick (`) characters for multiline string literals, likely borrowing syntax from other languages like JavaScript or Python. AL does not support backtick-delimited strings. In AL, multiline text must be constructed either using string concatenation with CR/LF characters, or using the newer AL multiline string literal syntax (available in newer BC versions) which uses single quotes with line breaks inside them. The compilation errors at line 13 with 'Unexpected character `' clearly indicate the model used backtick syntax which is invalid in AL.

Correct Pattern:
Use AL string concatenation with CR/LF characters, e.g.:
var CrLf: Text[2];
begin
  CrLf[1] := 13;
  CrLf[2] := 10;
  exit('SELECT CustomerNo, Name, Balance' + CrLf + 'FROM Customer' + CrLf + 'WHERE Active = true' + CrLf + 'ORDER BY Name');
end;

Or if the runtime supports it, use the AL multiline text constant syntax with single quotes spanning multiple lines.
Incorrect Pattern:
` (backtick-based multiline string literal attempt)

Error Codes: AL0183

2 inherent-permissions-syntax codeunit-properties 1 CG-AL-H019

Description: The model generated incorrect values for InherentEntitlements and InherentPermissions properties. The task specifies 'X' for both, which means Execute. However, the model likely wrote something like 'codeunit "CG Internal Service" = X' or used 'codeunit' as a permission value instead of the correct syntax. The valid values for InherentEntitlements and InherentPermissions are permission sets like 'X' (Execute), 'R' (Read), 'RIMD' etc., not object type references. The AL0776 error 'The identifier codeunit is not a valid permission value' indicates the model confused the property syntax, possibly writing something like 'InherentPermissions = codeunit 70019 = X' instead of simply 'InherentPermissions = X'.

Correct Pattern:
InherentEntitlements = X;
    InherentPermissions = X;
Incorrect Pattern:
InherentEntitlements = codeunit = X
    InherentPermissions = codeunit = X

Error Codes: AL0776

3 query-crossjoin-column-datasource query-object-definition 1 CG-AL-H017

Description: The model failed to generate any valid code (generated code not found), and when it did attempt generation, it produced a query where a Column did not have a valid data source or the Method property set to Count. In AL query objects with CrossJoin, the second dataitem is not nested under the first but is at the same level, and columns must correctly reference their parent dataitem's fields. The model likely struggled with the complex query structure involving CrossJoin between two instances of the same table ('Dimension Value'), proper column-to-dataitem binding, and the LeftOuterJoin with a Count method column. The AL0353 error at line 42 indicates a column was defined without properly referencing a source field from its parent dataitem.

Correct Pattern:
query 70017 "CG Dimension Matrix"
{
    QueryType = Normal;

    elements
    {
        dataitem(DeptDimValue; "Dimension Value")
        {
            SqlJoinType = CrossJoin;
            DataItemTableFilter = "Dimension Code" = const('DEPARTMENT');
            column(DepartmentCode; "Code") { }
            column(DepartmentName; Name) { }
            dataitem(ProjDimValue; "Dimension Value")
            {
                SqlJoinType = CrossJoin;
                DataItemTableFilter = "Dimension Code" = const('PROJECT');
                column(ProjectCode; "Code") { }
                column(ProjectName; Name) { }
                dataitem(DimSetEntry; "Dimension Set Entry")
                {
                    SqlJoinType = LeftOuterJoin;
                    DataItemLink = "Dimension Code" = DeptDimValue."Dimension Code";
                    column(MatchCount; "Entry No.")
                    {
                        Method = Count;
                    }
                }
            }
        }
    }
}
Incorrect Pattern:
// Generated code not found - model failed to produce valid output

Error Codes: AL0353

4 complete-codeunit-generation recordref-fieldref-dynamic-manipulation 1 CG-AL-H022

Description: The model failed to generate any valid AL code at all. The 'Generated code not found' note and the compilation error at line 60 (syntax error expecting '}') suggest the model either produced empty/malformed output or failed to produce a complete codeunit. The task required creating a complex codeunit with 8 procedures using RecordRef/FieldRef patterns, error handling with TryFunction-like patterns, and dynamic record manipulation. The model was unable to produce syntactically valid AL code for this complex task.

Correct Pattern:
codeunit 70224 "CG Dynamic Record Handler"
{
    Access = Public;

    procedure GetTableName(TableId: Integer): Text
    var
        RecRef: RecordRef;
    begin
        if not TryOpenTable(RecRef, TableId) then
            exit('');
        exit(RecRef.Name);
    end;

    // ... (complete implementation of all 8 procedures using RecordRef, FieldRef, KeyRef)
}
Incorrect Pattern:
// Generated code not found

Error Codes: AL0104

5 yaml-parsing-string-manipulation text-parsing-and-json-handling 1 CG-AL-M021

Description: The model generated AL code with multiple syntax errors including semicolon issues (AL0111) at various lines, likely from incorrect use of string operations or variable declarations, misuse of 'Key' as a variable name (AL0519 - reserved word), and general structural syntax errors. The generated code was not captured but the compilation errors indicate the model struggled with: (1) proper AL syntax for text splitting/parsing operations, (2) using 'Key' as a variable name which is reserved in AL, (3) proper foreach/iteration syntax over JSON objects, and (4) general AL procedure structure. The task and tests are valid - this is a complex string parsing task that requires manual YAML parsing since AL has no built-in YAML library, and the model failed to produce syntactically correct AL code.

Correct Pattern:
For YAML parsing in AL: split text by line feed character (char 10), then split each line by ': ' to get key-value pairs, store in JsonObject. Use variable names like 'KeyName' or 'PropertyKey' instead of reserved word 'Key'. For iterating JsonObject keys, use 'foreach PropertyName in JsonObj.Keys do' with proper AL syntax. Ensure all statements end with semicolons and proper begin/end blocks.
Incorrect Pattern:
// Generated code not available but errors indicate issues at lines 39, 54, 93, 104, 132 (semicolon expected) and line 143 using 'Key' as identifier

Error Codes: AL0111

6 parse-failure unknown 1 CG-AL-E052

Description: Failed to parse LLM analysis response: Looking at the failure details: 1. The generated code compiled successfully and most tests passed 2. Only `TestDateToText_ValidDate` and `TestDateToText_FirstDayOfYear` failed 3. The test for `TestDa

Correct Pattern:
Incorrect Pattern:
7 multiline-string-literals text-literal-syntax 1 CG-AL-E050

Description: The model attempted to use actual multiline string literals (spanning multiple lines in source code) which is not valid AL syntax in most BC versions. The AL0360 errors ('Text literal was not properly terminated') indicate the model wrote string literals that span across lines without proper termination. In AL, multiline text must be constructed using string concatenation with CR/LF characters, or using the newer AL multiline string literal syntax (triple-quoted strings) if supported. The model likely wrote something like 'SELECT CustomerNo, Name, Balance\nFROM Customer...' as a raw multiline string in the source, which the compiler rejected. The task explicitly asked for multiline string literals, but the model didn't know the correct AL syntax for achieving this (e.g., using concatenation with carriage return/line feed characters, or the proper triple-single-quote syntax for multiline literals if the runtime version supports it).

Correct Pattern:
In AL, multiline text can be achieved via concatenation: exit('SELECT CustomerNo, Name, Balance' + CrLf + 'FROM Customer' + CrLf + 'WHERE Active = true' + CrLf + 'ORDER BY Name'); where CrLf is built from char values 13 and 10. Alternatively, if the runtime supports it (runtime 12.0+), use triple-quoted multiline string literals with the syntax: '''...multiline content...'''
Incorrect Pattern:
exit('SELECT CustomerNo, Name, Balance
FROM Customer
WHERE Active = true
ORDER BY Name');

Error Codes: AL0360

8 codeunit-syntax-structure codeunit-definition 1 CG-AL-M005

Description: The model generated AL code with syntax errors - specifically missing 'end', ';', and '}' tokens at line 106. This indicates the model failed to properly structure the codeunit with correct AL syntax, likely producing malformed procedure bodies, incomplete begin/end blocks, or incorrect nesting. The generated code was not found/empty in the provided output, but the compilation errors at a specific line (106) with multiple syntax expectations suggest the model produced code that was structurally incomplete or had mismatched begin/end pairs. This is a fundamental AL syntax knowledge gap where the model couldn't produce a syntactically valid codeunit with multiple procedures involving HttpClient, JsonObject, and error handling patterns.

Correct Pattern:
codeunit 70002 "External Payment Service"
{
    procedure SendPaymentRequest(OrderId: Text; Amount: Decimal; Currency: Text; var ResponseJson: JsonObject): Boolean
    var
        Client: HttpClient;
        RequestMessage: HttpRequestMessage;
        ResponseMessage: HttpResponseMessage;
        Content: HttpContent;
        RequestJson: JsonObject;
        ResponseText: Text;
    begin
        // Build request, send via HttpClient, parse response
        // Proper begin/end blocks with error handling
        exit(false);
    end;

    procedure ValidatePaymentResponse(Response: JsonObject): Boolean
    var
        Token: JsonToken;
        StatusText: Text;
    begin
        if not Response.Get('status', Token) then
            exit(false);
        if not Response.Get('transactionId', Token) then
            exit(false);
        if not Response.Get('amount', Token) then
            exit(false);
        Response.Get('status', Token);
        StatusText := Token.AsValue().AsText();
        exit(StatusText = 'approved');
    end;

    // ... remaining procedures with proper begin/end structure
}
Incorrect Pattern:
// Generated code not available but compilation failed at line 106 with syntax errors expecting 'end', ';', and '}'

Error Codes: AL0104

9 complex-report-with-helper-codeunit report-definition-and-codeunit-generation 1 CG-AL-M007

Description: The model failed to generate any valid AL code at all. The task required creating both a Report 70001 'Sales Performance Analysis' and a helper codeunit 'CG-AL-M007 Mock Calculator' that the test file references. The generated code appears to be empty or syntactically broken (compilation error at line 31 with 'Syntax error, } expected' and 'App generation failed'). The model did not understand how to produce a complete solution consisting of: (1) a report object with multiple data items (Customer, Sales Header, Sales Line), request page, and triggers, and (2) a mock calculator codeunit with procedures like Initialize(), AddSalesLine(), GetRunningTotalByCustomer(), GetRunningTotalByRegion(), CalculateAverageOrderValue(), GetCustomerRank(), GetTopProduct(), GetProductSalesQuantity(), CalculateYoYComparison(), CalculateOrderFrequency(), GetTotalSales(), and GetCustomerCount(). The model either produced no output or produced fundamentally broken AL syntax.

Correct Pattern:
The solution requires at minimum two AL objects:

1. report 70001 "Sales Performance Analysis" with DataItems for Customer, Sales Header, Sales Line, request page with date filters, and proper ApplicationArea/UsageCategory.

2. codeunit XXXXX "CG-AL-M007 Mock Calculator" with procedures:
   - Initialize()
   - AddSalesLine(CustomerCode: Code[20]; Region: Code[20]; ItemCode: Code[20]; Quantity: Decimal; Amount: Decimal)
   - GetRunningTotalByCustomer(CustomerCode: Code[20]): Decimal
   - GetRunningTotalByRegion(Region: Code[20]): Decimal
   - CalculateAverageOrderValue(): Decimal
   - GetCustomerRank(CustomerCode: Code[20]): Integer
   - GetTopProduct(): Code[20]
   - GetProductSalesQuantity(ItemCode: Code[20]): Decimal
   - CalculateYoYComparison(CurrentYear: Decimal; PreviousYear: Decimal): Decimal
   - CalculateOrderFrequency(OrderCount: Integer; Days: Integer): Decimal
   - GetTotalSales(): Decimal
   - GetCustomerCount(): Integer

Using temporary record variables or Dictionary collections to store and aggregate the in-memory sales data.
Incorrect Pattern:
// Generated code not found (empty or completely invalid output causing syntax error at line 31)

Error Codes: AL0104