← Back to Benchmark Results

openrouter/x-ai/grok-code-fast-1

57.1%
Pass Rate
32/56
Tasks Passed
3
Runs
52.4%
pass@1
57.1%
pass@3
89.3%
Consistency
0.1
Temperature
-
Thinking
1,238,933
Tokens
$10.77
Cost
1st: 682nd: 20Failed: 2432/56 passed

Known Shortcomings (12)

Sorted by occurrence count (most frequent first)

# Concept AL Concept Count Affected Tasks
1 query-object-syntax query-definition 2 CG-AL-H011, CG-AL-H017

Description: The model failed to generate valid AL code for a query object. The compilation errors (AL0107 'identifier expected' at line 27:32, AL0104 '=' expected, etc.) indicate the model produced syntactically invalid AL for the query definition. The generated code was either empty or had fundamental syntax errors in the query structure, likely around the filter element or column definitions. The model doesn't properly know how to construct an AL query object with dataitem, columns with Method = Sum/Count, filter elements with ColumnFilter, and OrderBy clauses.

Correct Pattern:
query 70011 "CG Sales Summary"
{
    QueryType = Normal;
    OrderBy = ascending(Sell_to_Customer_No);

    elements
    {
        dataitem(SalesLine; "Sales Line")
        {
            column(Document_No; "Document No.") { }
            column(Sell_to_Customer_No; "Sell-to Customer No.") { }
            column(Line_Amount_Sum; "Line Amount") { Method = Sum; }
            column(Line_Count; "Line Amount") { Method = Count; }
            filter(Document_Type; "Document Type") { ColumnFilter = const(Order); }
        }
    }
}
Incorrect Pattern:
// Generated code not found - but compilation errors at line 27 suggest the model produced a query with syntax errors around filter/column definitions

Error Codes: AL0107

2 multiline-string-literals string-literals 1 CG-AL-E050

Description: The model failed to correctly use AL multiline string literals. The compilation errors (AL0361 'Identifier was not properly terminated', syntax errors about semicolons and expected tokens on consecutive lines) strongly indicate the model attempted to use a multiline string syntax that is not valid in AL. In AL, multiline text literals were introduced in later runtime versions and use a specific syntax (text enclosed in single quotes with actual line breaks, or using TextBuilder/string concatenation). The model likely tried to use double-quoted multiline strings or some other language's multiline syntax (e.g., Python triple quotes or C# verbatim strings), which caused the parser to fail at line 8-11 of the generated file. The task specifically requires multiline string literals, which in AL (runtime 11.0+) use single-quote delimited strings that span multiple lines.

Correct Pattern:
In AL (runtime 11.0+), multiline string literals use single quotes spanning multiple lines, e.g.:

procedure GetSqlQuery(): Text
begin
    exit(
        'SELECT CustomerNo, Name, Balance' + Environment.NewLine() +
        'FROM Customer' + Environment.NewLine() +
        'WHERE Active = true' + Environment.NewLine() +
        'ORDER BY Name'
    );
end;

Or if the runtime supports it, use the actual multiline text constant syntax with single quotes.
Incorrect Pattern:
The generated code was not captured, but based on errors at lines 8-11 involving unterminated identifiers and missing semicolons, the model likely wrote something like: exit("SELECT CustomerNo, Name, Balance\nFROM Customer\nWHERE Active = true\nORDER BY Name"); using double quotes or an invalid multiline syntax.

Error Codes: AL0361

3 page-extension-cardpageid-override page-extension-properties 1 CG-AL-E053

Description: The model failed to generate valid AL code for a page extension. The compilation errors at line 3 and line 15 suggest the model produced syntactically invalid code, likely struggling with the BC 2025 Wave 1 feature of overriding CardPageId on a list page extension. The errors indicate fundamental syntax problems - missing semicolons, unexpected tokens, and unrecognized object keywords - meaning the model either produced malformed page extension syntax or attempted to use an incorrect syntax for the CardPageId property override. The correct pattern requires a standard pageextension declaration with the CardPageId property set at the extension level, plus an action in the actions section.

Correct Pattern:
pageextension 70053 "CG Item List Extension" extends "Item List"
{
    CardPageId = "Item Card";

    actions
    {
        addlast(Processing)
        {
            action("Show Item Details")
            {
                ApplicationArea = All;
                Caption = 'Show Item Details';
                Image = Card;
                RunObject = page "Item Card";
                RunPageLink = "No." = field("No.");
            }
        }
    }
}
Incorrect Pattern:
// Generated code not found - but errors at line 3:22 and 15:25 suggest malformed pageextension declaration with incorrect syntax for CardPageId override

Error Codes: AL0104

4 json-api-methods json-object-manipulation 1 CG-AL-H014

Description: The model generated code with multiple errors related to AL's JsonObject and JsonToken API usage: (1) AL0133 - Tried to pass a Text to a Boolean parameter, likely misusing a method overload. (2) AL0135 - Called JsonObject.Get(Text) without the required second 'var JsonToken' parameter - the correct signature is Get(Text, var JsonToken). (3) AL0132 - Tried to call 'AsDecimal' on JsonToken, but JsonToken doesn't have AsDecimal directly; you need to convert to JsonValue first via AsValue().AsDecimal(). The task description mentions 'typed JSON getter methods' like GetText(), GetInteger(), GetBoolean(), GetArray() which don't exist as direct methods on JsonObject in AL. The model needed to use the pattern: JsonObject.Get('key', JsonToken) then JsonToken.AsValue().AsText()/AsInteger()/AsDecimal() etc. The task description itself is somewhat misleading by referencing methods like GetText('name') that don't exist in AL's JSON API, but the model should have known the correct AL JSON patterns regardless.

Correct Pattern:
var JToken: JsonToken; JValue: JsonValue; begin CustomerJson.Get('name', JToken); JValue := JToken.AsValue(); NameText := JValue.AsText(); // For nested: RootJson.Get('data', JToken); DataObj := JToken.AsObject(); DataObj.Get('details', JToken); DetailsObj := JToken.AsObject(); DetailsObj.Get('amount', JToken); exit(JToken.AsValue().AsDecimal());
Incorrect Pattern:
Not available - generated code file was not found, but errors indicate incorrect JsonObject.Get() calls without var JsonToken parameter, incorrect boolean conversion, and JsonToken.AsDecimal() usage

Error Codes: AL0133

5 recordref-fieldref-dynamic-manipulation RecordRef-FieldRef-usage 1 CG-AL-H022

Description: The model generated code with syntax errors (semicolon expected at multiple lines around 101, 104, 128, 131). The generated code was not captured in the output, but the compilation errors indicate the model produced invalid AL syntax in the codeunit implementation. The errors at lines 101, 104, 128, 131 suggest issues in procedure bodies - likely incorrect statement termination, possibly using incorrect syntax for try/catch patterns (e.g., using 'try' or 'catch' keywords incorrectly), or malformed if-then-else blocks. The task and tests are valid; the model simply failed to produce syntactically correct AL code for the dynamic record handler codeunit.

Correct Pattern:
In AL, error handling uses [TryFunction] attribute on procedures or if not RunResult := Codeunit.Run() pattern. There is no try/catch syntax. Statements must end with semicolons. Example for safe RecordRef.Open: procedure GetTableName(TableId: Integer): Text; var RecRef: RecordRef; begin if not TryOpenTable(RecRef, TableId) then exit(''); exit(RecRef.Name); end; [TryFunction] local procedure TryOpenTable(var RecRef: RecordRef; TableId: Integer) begin RecRef.Open(TableId); end;
Incorrect Pattern:
// Generated code not captured, but compilation errors at lines 101, 104, 128, 131 indicate semicolon-expected errors, likely from incorrect error handling syntax or malformed control flow statements

Error Codes: AL0111

6 interface-and-implementation-syntax interface-definition 1 CG-AL-M009

Description: The model generated AL code with syntax errors at line 100, likely involving incorrect return type syntax or procedure signatures in the interface/implementation. The compilation errors (expecting ')', 'end', and semicolons at the same line) suggest the model wrote malformed procedure signatures, possibly using incorrect syntax for return types (e.g., Text[50] as a return type in a procedure declaration) or other structural issues in the codeunit implementing the interface. The task and tests are valid - the test file uses a separate mock codeunit and simply validates the interface contract. The model failed to produce syntactically correct AL code for the interface definition and/or its implementation.

Correct Pattern:
interface "Shipping Provider"
{
    procedure CalculateShippingCost(Weight: Decimal; FromCountry: Text; ToCountry: Text): Decimal;
    procedure EstimateDeliveryTime(FromCountry: Text; ToCountry: Text; ServiceType: Text): Integer;
    procedure CreateShipment(OrderNumber: Text; FromAddress: Text; ToAddress: Text; Weight: Decimal): Text[50];
    procedure TrackShipment(TrackingNumber: Text): Text[100];
    procedure ValidateAddress(Street: Text; City: Text; State: Text; ZipCode: Text; Country: Text): Boolean;
}

codeunit 70004 "Standard Shipping Provider" implements "Shipping Provider"
{
    // ... proper implementation of all interface methods
}
Incorrect Pattern:
Generated code not available but line 100 has syntax errors suggesting malformed procedure signature, likely something like: procedure CreateShipment(...): Text[50] with incorrect syntax

Error Codes: AL0104

7 jsonobject-get-returns-jsontoken json-handling 1 CG-AL-M021

Description: The model generated code that passes Text variables directly as the second argument to JsonObject.Get(), but JsonObject.Get() requires a 'var JsonToken' parameter. The model should have used a JsonToken variable, then extracted the text value from the token using AsValue().AsText(). Additionally, the model tried to call JsonObject.Set() which doesn't exist - the correct method is Replace() for existing keys or Add() for new keys.

Correct Pattern:
var JToken: JsonToken;
JsonObj.Get('name', JToken);
NameValue := JToken.AsValue().AsText();
// For setting values, use JsonObj.Replace(Key, Value) or JsonObj.Add(Key, Value)
Incorrect Pattern:
JsonObj.Get('name', NameValue); // where NameValue is Text
JsonObj.Set(...);

Error Codes: AL0133

8 jsonobject-selecttoken-vs-get json-api-usage 1 CG-AL-M020

Description: The model incorrectly used JsonObject.Get() by passing typed variables (Text, Decimal, Boolean, Integer, JsonArray) directly as the second argument, but JsonObject.Get() expects a 'var JsonToken' as the second parameter. The correct pattern is to first get a JsonToken, then extract the value from the JsonToken via AsValue().AsText(), AsValue().AsDecimal(), AsArray(), etc. The model also tried to use 'foreach' on a JsonArray directly, which is not supported — you must iterate using an index-based loop with JsonArray.Get(Index, JsonToken). These are fundamental AL JSON API patterns the model failed to apply correctly.

Correct Pattern:
var JToken: JsonToken; JValue: JsonValue;
ProductJson.Get('name', JToken);
JValue := JToken.AsValue();
NameText := JValue.AsText();

ProductJson.Get('price', JToken);
PriceDecimal := JToken.AsValue().AsDecimal();

// For arrays:
ProductJson.Get('values', JToken);
ValuesArray := JToken.AsArray();
for i := 0 to ValuesArray.Count - 1 do begin
  ValuesArray.Get(i, JToken);
  Sum += JToken.AsValue().AsInteger();
end;
Incorrect Pattern:
ProductJson.Get('name', NameText); ProductJson.Get('price', PriceDecimal); ProductJson.Get('inStock', InStockBool); ProductJson.Get('quantity', QuantityInt);

Error Codes: AL0133

9 string-numeric-extraction-and-padding text-manipulation-number-formatting 1 CG-AL-E051

Description: The model generated code that fails to properly extract the numeric portion from a string, increment it, and re-insert it with the original zero-padding preserved. The error 'Expected:<DOC-002> Actual:<DOC- 2>' shows the model is replacing the numeric portion with a space-padded number instead of zero-padded. The model likely used Format() or a similar function that produces space-padded output instead of properly using PadStr or StrSubstNo with zero-padding, or manually constructing the zero-padded string. The model needs to: 1) Find the trailing numeric portion of the string, 2) Record its length for zero-padding, 3) Parse it as an integer, 4) Increment/decrement, 5) Format the result back with leading zeros to match the original width, 6) Concatenate with the prefix.

Correct Pattern:
To zero-pad in AL, use PadStr('', NumLen - StrLen(Format(NewNum)), '0') + Format(NewNum), or manually build the zero-padded string. For example: NumText := Format(NewNum); while StrLen(NumText) < OriginalNumLen do NumText := '0' + NumText; Result := Prefix + NumText;
Incorrect Pattern:
// Generated code not found - but based on error output, the model produced code that formats numbers with space padding instead of zero padding, e.g., Format(NewNum, NumLen) which produces '  2' instead of '002'
10 foreach-loop-continue-pattern loop-constructs-and-list-iteration 1 CG-AL-H013

Description: The model generated incorrect code for the SumPositiveNumbers procedure. Based on the test output, the sum returned 30 instead of 60 for inputs [10, 20, 30], suggesting the model's implementation only summed the last element or had a logic error such as reassigning instead of accumulating (e.g., using ':=' instead of '+='). The generated code was not captured but all tests failed, indicating fundamental implementation errors across all three procedures. The model failed to correctly implement foreach loops with continue statements, proper accumulation patterns, and comma-separated string building with arrays.

Correct Pattern:
procedure SumPositiveNumbers(Numbers: List of [Decimal]): Decimal
var
    Number: Decimal;
    Sum: Decimal;
begin
    foreach Number in Numbers do begin
        if Number <= 0 then
            continue;
        Sum += Number;
    end;
    exit(Sum);
end;
Incorrect Pattern:
// Generated code not found - but based on error 'Expected:<60> Actual:<30>' for inputs [10,20,30], the model likely wrote something like: Sum := Number; instead of Sum := Sum + Number; or had an off-by-one/logic error in the loop
11 httpclient-getheaders-usage http-client-api 1 CG-AL-M005

Description: The model generated code that calls GetHeaders() without the required HttpHeaders parameter. In AL, HttpRequestMessage.GetHeaders(var HttpHeaders) and HttpContent.GetHeaders(var HttpHeaders) require an HttpHeaders variable to be passed as a VAR parameter. The model likely wrote something like `Request.GetHeaders().Add(...)` instead of declaring an HttpHeaders variable and passing it: `Request.GetHeaders(Headers); Headers.Add(...)`. This is a common pattern misunderstanding with the AL HttpClient API where GetHeaders is not a function that returns headers directly but rather populates a VAR parameter.

Correct Pattern:
var Headers: HttpHeaders;
...
Request.GetHeaders(Headers);
Headers.Add('Authorization', 'Bearer ' + AuthToken);
Incorrect Pattern:
Request.GetHeaders().Add(...) or similar calls without passing the required HttpHeaders VAR parameter

Error Codes: AL0135

12 complex-report-with-mock-codeunit report-definition-and-helper-codeunit 1 CG-AL-M007

Description: The model failed to generate valid AL code for a complex task requiring both a Report object (Sales Performance Analysis, ID 70001) and a helper codeunit (CG-AL-M007 Mock Calculator) used by the tests. The compilation errors at line 274 indicate a syntax error in the generated code - likely the model produced malformed AL syntax (possibly mixing object definitions incorrectly, using wrong delimiters, or failing to properly structure one of the required objects). The errors AL0104 ('Syntax error, ':' expected', 'Syntax error, ';' expected') and AL0198 ('Expected one of the application object keywords') suggest the model produced code that broke the AL parser, possibly by including invalid syntax within a procedure body or object definition, or by failing to properly close an object before starting another one.

Correct Pattern:
The task requires generating: 1) A Report 70001 'Sales Performance Analysis' with Customer, Sales Header, and Sales Line data items, request page with date filters, and proper triggers; 2) A Codeunit 'CG-AL-M007 Mock Calculator' with procedures: Initialize(), AddSalesLine(CustomerCode, Region, ItemCode, Quantity, Amount), GetRunningTotalByCustomer(CustomerCode): Decimal, GetRunningTotalByRegion(Region): Decimal, CalculateAverageOrderValue(): Decimal, GetCustomerRank(CustomerCode): Integer, GetTopProduct(): Text, GetProductSalesQuantity(ItemCode): Decimal, CalculateYoYComparison(CurrentYear, PreviousYear): Decimal, CalculateOrderFrequency(OrderCount, Days): Decimal, GetTotalSales(): Decimal, GetCustomerCount(): Integer. All objects must have proper AL syntax with correct object declarations, variable sections, and procedure definitions.
Incorrect Pattern:
Generated code not available (line 274 area has syntax errors suggesting malformed variable declaration, procedure definition, or object structure)

Error Codes: AL0104