Insurance
Canadian insurance professionals label documents, footage and decisions, and test the models that act on them, in English and French.
Four lines of insurance work, each with a public benchmark or dataset to start from and a Canadian layer our experts add.
Policy wording, claim forms, estimates and correspondence labelled by field and linked to source passages, in both languages. Evaluations check extraction, coverage interpretation and escalation when information is missing.
Road footage and in-cab video labelled for objects, events and driver behaviour, from a safe stop to distraction. Used for rewards programmes, claims evidence and behaviour models.
Appetite, qualification, limits and product recommendation tasks over guidelines and application data. Experts score the decision and the reasoning.
Questions and scenarios under Quebec civil law and the certification material for insurance representatives, in French. The gap between English and French performance is measured, not assumed.
Claims documents
English and French insurance document fields are located, extracted and aligned to one structured record. Experts link every extracted value to its source passage and flag what the document does not support.
Insurance document processingDriving footage
Rewards programmes and claims evidence depend on what a driver did, not only what the camera saw. Annotators label events, interactions and outcomes with timestamps, and evaluate model descriptions against the footage.
Video annotationPublished results for open-weight models you can run yourself. Our Canadian work adds Quebec civil law, bilingual documents and the products Canadian insurers actually sell.
807 multiple-choice questions in French, digitized from the manuals used to certify insurance representatives in Quebec, under Quebec civil law.
| Model | Closed-book | Closed-book | With retrieval |
|---|---|---|---|
| DeepSeek-R1 (deepseek-reasoner)Open weightsDeepSeek · MIT licence | 36.30% | 71.77% | |
| R1-1776Open weightsPerplexity · MIT licence | 49.38% | 69.09% | |
| DeepSeek-V3 (deepseek-chat)Open weightsDeepSeek · MIT licence | 58.23% | 67.78% | |
| Qwen3-30B-A3BOpen weightsAlibaba · Apache 2.0 | 54.77% | 37.78% | |
| QwQ-32BOpen weightsAlibaba · Apache 2.0 | 51.36% | 27.49% | |
| Llama-3.3-70B-InstructOpen weightsMeta · Llama licence | 61.48% | 0.99% | |
| Llama-3.3-Nemotron-Super-49BOpen weightsNVIDIA · open weights | 47.20% | 3.09% | |
| Granite-3.2-8B-InstructOpen weightsIBM · Apache 2.0 | 30% | 30.86% |
Commercial property and casualty underwriting: appetite, qualification, limits and deductibles, product recommendation, business classification and small-business eligibility, using SQL and guideline tools over a database and proprietary business rules.
| Model | Accuracy | Accuracy |
|---|---|---|
| DeepSeek V3.1Open weightsDeepSeek · MIT licence | 73.70% | |
| Qwen3 Coder 480BOpen weightsAlibaba · Apache 2.0 | 73.30% | |
| Kimi K2 InstructOpen weightsMoonshot AI · modified MIT | 56.70% | |
| Qwen3 235BOpen weightsAlibaba · Apache 2.0 | 30% | |
| gpt-oss-120bOpen weightsOpenAI · Apache 2.0 | 30% |
Public driving and insurance-vision datasets set the baseline for object and behaviour labels. Our footage work adds Canadian roads, seasons and signage, and the behaviour labels an adjuster needs.
10 classes, from safe driving to texting, phone use, reaching behind, drinking and talking to a passenger.
State Farm · Kaggle, 201610 perception tasks: object boxes, drivable area, lane markings, segmentation and more; crowdsourced from over 50,000 rides.
UC Berkeley22 fundamental, 12 meta and 5 scenario tasks for vision-language models; 11 models evaluated.
ICCV 2025Document-grounded cases written with practising underwriters and claims handlers, scored pass@1. Leaderboard opening 2026.
Huzzle LabsFour suites including banking. Published open-weight result: Llama 3 70B completes 34.02% of tasks, 18.28% under attack; 25.60% of injections succeed.
ETH Zurich · NeurIPS 2024RL environments in Canada
Simulated claims, renewal and underwriting tasks where a model acts on a policyholder’s behalf and an expert-written reward scores what it did. Built on Quebec and common-law policy wording, in English and French.
File a claim for the hail damage from last Thursday.
Your claim is open.
Tell us which line of business and which models you are evaluating.