Developer evaluation guide

Gemini 4 Argon coding test

From published benchmarks to code you can verify. Inspect the evidence and evaluation conditions, then try four focused programming tasks.

Sources checked:

What the published scores measure

Google reports 77.9% on DeepSWE v1.1. Its launch table also lists 55.0% on FrontierSWE v2 and 57.4% on Terminal-Bench 4.0. These use different task sets, not interchangeable measures of coding quality.

Separate 0–100% bars for Argon: DeepSWE v1.1 77.9%, FrontierSWE v2 55.0%, Terminal-Bench 4.0 57.4%.
Each bar describes its own benchmark, not a model ranking or overall score. Source: Google launch table. Original score table
Gemini 4 Argon · September/October 2026 release snapshot
BenchmarkPublished scoreInterpretation
DeepSWE v1.177.9%Long-horizon repository work; read the agent harness and trial policy.
FrontierSWE v255.0%Another engineering task set; its percentage is not directly comparable with DeepSWE.
Terminal-Bench 4.057.4%Terminal tasks; environment and tool access affect results.

Compare conditions before comparing models

For DeepSWE v1.1, Google used mini-swe-agent for Argon. Its methodology cites leaderboard results for GPT and system cards for Claude, using the highest reported thinking levels. Identical harnesses and budgets are not established.

Other model records · different provenance, not a controlled head-to-head test
ModelProvenance / comparability
Gemini 3.8 FlashNot published in the cited sources. No matching result for Gemini 3.8 Flash in this release table. Do not substitute Flash Cyber or another version.
GPT-6 AstraDeepSWE v1.1: 74.1%, quoted by Google from the leaderboard. Equal harnesses and budgets with Argon are unverified.
Claude Opus 5.5DeepSWE v1.1: 74.2%, quoted by Google from a system card. Equal harnesses and budgets with Argon are unverified.

Historical published records do not measure today’s API or establish a winner for every coding workflow.

A repeatable workflow

Use one first attempt per model per task. Freeze prompts and checks; report repairs separately. Record the exact model ID, date, thinking level, tools, runtime, dependency versions, elapsed time, token usage, exit code and unedited output. Mark unavailable fields as unknown.

Five steps: prompt, generated code, execute, assertions, record evidence.
Original workflow illustration. Expected output is a test oracle; observed output must come from an actual run.

Four reproducible programming tasks

Use a fresh conversation per task. Keep prompts, tools, dependencies and attempt budgets identical across models. Save the supplied checks before generating code. Run generated code in a disposable local directory.

01 · TypeScript concurrency pool

Awaiting model test

Model output and observed result: not available; no model run has been performed. These checks are authored fixtures, not model output.

Prompt to give the model

Implement pool.ts exporting async function pool<T>(tasks: Array<() => Promise<T>>, limit: number): Promise<PromiseSettledResult<T>[]>. Start tasks lazily, run at most limit simultaneously, keep input order, settle promise rejection and synchronous throws without stopping other tasks. Empty input returns []. Reject non-positive/non-integer limits with RangeError even for empty input. Use no dependencies. Return only pool.ts. Do not modify the supplied checks.

Input / starter files

// Input: delays [30, 5, 15] ms; limit 2.
// Implementation file: pool.ts

Save these checks before testing

// Save as pool.check.ts (Node 22+)
import assert from 'node:assert/strict';
import { pool } from './pool.ts';
let active = 0, peak = 0;
const tasks = [30, 5, 15].map((ms, i) => async () => {
 active++; peak = Math.max(peak, active);
 await new Promise(r => setTimeout(r, ms));
 active--; return i;
});
assert.deepEqual(await pool(tasks, 2), [0,1,2].map(value => ({status:'fulfilled', value})));
assert.equal(peak, 2);
const rejected = await pool([() => { throw new Error('sync'); }, () => Promise.reject('async'), async () => 7], 1);
assert.deepEqual(rejected.map(r => r.status), ['rejected','rejected','fulfilled']);
assert.deepEqual(rejected[2], {status:'fulfilled', value:7});
assert.deepEqual(await pool([], 2), []);
for (const limit of [0, -1, 1.5, NaN, Infinity]) {
 await assert.rejects(async () => pool([], limit), RangeError);
}
console.log('pool: all assertions passed');

Run locally

npx --yes --package tsx@4.21.0 tsx pool.check.ts

Expected behavior / acceptance

Results remain [0, 1, 2], peak concurrency equals 2, and rejection does not stop later tasks. Empty input and invalid limits pass. Expected final line: pool: all assertions passed. Stop a hung run after 30 seconds and record a timeout.

02 · React search race repair

Awaiting model test

Model output and observed result: not available; no model run has been performed. These checks are authored fixtures, not model output.

Prompt to give the model

Fix useSearch.ts below. Keep the API useSearch(query, search), where search(q) returns Promise<string[]>. Only the latest query may update state. Clear results and error for blank/whitespace queries; suppress stale success AND rejection; set error to the current rejection message; suppress updates after unmount. search is stable. Return an object with items and error fields; error is initially null. Return only useSearch.ts; do not change checks.

Input / starter files

// Save as useSearch.ts — intentionally broken starter
import { useEffect, useState } from 'react';
export function useSearch(query: string, search: (q: string) => Promise<string[]>) {
 const [items, setItems] = useState<string[]>([]);
 const [error, setError] = useState<string | null>(null);
 useEffect(() => { search(query).then(setItems).catch(e => setError(e.message)); }, [query, search]);
 return {items, error};
}

Save these checks before testing

// Save as race.check.test.tsx in a fresh npm directory.
// @vitest-environment jsdom
import { afterEach, expect, test, vi } from 'vitest';
import { act, cleanup, renderHook } from '@testing-library/react';
import { useSearch } from './useSearch';
afterEach(cleanup);
const deferred = () => {
 let resolve!: (v:string[])=>void;
 let reject!: (e:Error)=>void;
 const promise = new Promise<string[]>((a,b)=>{resolve=a;reject=b});
 return {promise,resolve,reject};
};
test('latest request, blank query, rejection and unmount', async () => {
 const a=deferred(), b=deferred(), c=deferred(), d=deferred();
 const pending: Record<string, ReturnType<typeof deferred>> = {a,b,c,d};
 const search=vi.fn((q:string) => pending[q].promise);
 const {result, rerender, unmount}=renderHook(({q})=>useSearch(q,search), {initialProps:{q:'a'}});
 rerender({q:'b'});
 await act(async()=>b.resolve(['B']));
 await act(async()=>a.resolve(['A']));
 expect(result.current.items).toEqual(['B']);
 rerender({q:'c'}); rerender({q:'  '});
 await act(async()=>c.reject(new Error('stale')));
 expect(result.current).toEqual({items:[],error:null});
 expect(search).toHaveBeenCalledTimes(3);
 rerender({q:'d'});
 await act(async()=>d.reject(new Error('offline')));
 expect(result.current.error).toBe('offline');
 const e=deferred();
 const next=()=>e.promise;
 const h=renderHook(()=>useSearch('e',next));
 h.unmount(); await act(async()=>e.resolve(['late']));
 unmount();
});

Run locally

npm init -y
npm install --save-exact react@19.2.1 react-dom@19.2.1
npm install --save-dev --save-exact vitest@3.2.4 jsdom@26.1.0 @testing-library/react@16.3.0
npx vitest run race.check.test.tsx

Expected behavior / acceptance

Resolve b before a; only B remains. Start c, clear query, then reject c; items stay empty and error stays null. Reject current d; error becomes offline. Vitest reports one passed test. Also review effect cleanup: an unmount assertion alone cannot prove updates were suppressed.

03 · Python CSV aggregation

Awaiting model test

Model output and observed result: not available; no model run has been performed. These checks are authored fixtures, not model output.

Prompt to give the model

Write summarize.py for Python 3.11+, standard library only. Read UTF-8 CSV from stdin: name,amount,currency. Sum Decimal amounts by currency, trim fields, uppercase currencies, accept only USD/EUR. Skip rows with missing/blank fields, unsupported currency, malformed/non-finite amounts or more than 2 decimal places. Support quoted commas in names. Output JSON with sorted currency keys and string values with two decimals, plus skipped count: totals (currency-to-amount object) and skipped (integer). Empty CSV gives empty totals and skipped 0. Do not use float. Return only summarize.py.

Input / starter files

name,amount,currency
"Doe, Jane",0.10,usd
Bob,0.20,USD
Eve,-1.00,EUR
X,NaN,USD
Y,2.345,USD
Z,10,GBP
Missing,,USD

Save these checks before testing

# Save as csv_check.py next to summarize.py
import json, subprocess, sys
sample = 'name,amount,currency\n"Doe, Jane",0.10,usd\nBob,0.20,USD\nEve,-1.00,EUR\nX,NaN,USD\nY,2.345,USD\nZ,10,GBP\nMissing,,USD\n'
def run(data):
 p = subprocess.run([sys.executable, 'summarize.py'], input=data, text=True, capture_output=True, timeout=5, check=True)
 return json.loads(p.stdout)
assert run(sample) == {'totals': {'EUR':'-1.00', 'USD':'0.30'}, 'skipped':4}
assert run('name,amount,currency\n') == {'totals':{}, 'skipped':0}
assert run('name,amount,currency\nA,Infinity,USD\n') == {'totals':{}, 'skipped':1}
print('csv: all assertions passed')

Run locally

python csv_check.py

Expected behavior / acceptance

Quoted Doe, Jane parses as one name. USD 0.10 + 0.20 becomes exactly "0.30"; EUR total is "-1.00"; 4 rows are skipped. Empty table and Infinity checks pass. Expected final line: csv: all assertions passed.

04 · Cross-file interface refactor

Awaiting model test

Model output and observed result: not available; no model run has been performed. These checks are authored fixtures, not model output.

Prompt to give the model

Refactor the three TypeScript files below. Replace displayName with firstName and lastName in internal User. Export getDisplayName(user): string from user.ts; trim names and join nonempty parts with one space. Update renderUser and toPublicUser, preserving their names, return types and public output object with id and name fields. Remove internal displayName references. No any, type assertions or dependencies. Return all three full files. Do not change checks.

Input / starter files

// user.ts
export type User = {id: string; displayName: string};
// view.ts
import type { User } from './user';
export const renderUser = (user: User): string => user.displayName;
// api.ts
import type { User } from './user';
export const toPublicUser = (user: User): {id: string; name: string} => ({id: user.id, name: user.displayName});

Save these checks before testing

// Save as refactor.check.ts
import assert from 'node:assert/strict';
import { getDisplayName, type User } from './user';
import { renderUser } from './view';
import { toPublicUser } from './api';
const u: User = {id:'u1', firstName:' Ada ', lastName:' Lovelace '};
assert.equal(getDisplayName(u), 'Ada Lovelace');
assert.equal(renderUser(u), 'Ada Lovelace');
assert.deepEqual(toPublicUser(u), {id:'u1', name:'Ada Lovelace'});
assert.equal(getDisplayName({...u, firstName:' ', lastName:' Ng '}), 'Ng');
assert.equal(getDisplayName({...u, firstName:'', lastName:''}), '');
console.log('refactor: all assertions passed');

Run locally

npx --yes --package typescript@5.9.3 tsc --noEmit --strict --target ES2022 --module ESNext --moduleResolution bundler user.ts view.ts api.ts
npx --yes --package tsx@4.21.0 tsx refactor.check.ts

Expected behavior / acceptance

Ada / Lovelace with surrounding spaces becomes Ada Lovelace in both consumers; API output stays id="u1", name="Ada Lovelace". Single-name and empty-name cases pass. Strict typecheck succeeds. Review the diff for any, assertions and leftover displayName references.

What is available at code level?

Google describes a libgav1 Rust optimization with identical video output and a 2.7× speedup over an earlier Rust port. The report does not supply a complete prompt, generated patch and execution transcript. We link the account without inventing a code excerpt or claiming this as our own result.

How to use this evidence

Our analysis: start with a small task from your own repository. For one-shot generation, check correctness and edge cases; for debugging, preserve a failing regression test; for longer tasks, inspect the full diff and compatibility. Published scores justify an evaluation but do not replace your acceptance checks.

Questions developers ask