Research on trustworthy and
explainable AI for Turkish law

Abstract

The study proposes the “Domain Classification” method to boost language-model performance in a field with its own terminology: the hypothesis that domain models trained on sub-corpora split by legal discipline outperform a single general model is tested experimentally. Four character-level GRU language models — commercial, labor, civil-procedure and constitutional law — were trained from scratch, coupled with a separate classifier RNN that detects the legal branch of the user's input and routes it to the matching language model, forming a hybrid natural language generation system.

Findings

• Domain models trained with the method improved test accuracy by 6.9%–13.6% over the general model, and all of them tracked below the general model in cross-entropy loss.
• A corpus of 577,748 sentences from 100 doctoral law theses, classified into 28 legal categories, was used for training.
• The legal-text classifier reached a macro F1 of 0.81 across five categories, peaking in commercial law (F1 = 0.86) and constitutional law (F1 = 0.82).
• The lightweight architecture, trainable even on low-end hardware, laid a foundation for legal text-generation scenarios such as drafting petitions, decisions and indictments.

Method details

The language models use Gated Recurrent Units (GRU): they avoid the vanishing/exploding-gradient issues of vanilla RNNs while training faster than LSTMs. A character-level representation was chosen for its effectiveness in morphologically rich, agglutinative languages such as Turkish (4 GRU layers, 512 units per layer, 256-character sequences, Log-Softmax activation, Adam optimizer). Beyond accuracy and cross-entropy loss, log-perplexity was computed as a language-model-specific criterion. The classifier is a word-level, 3-layer GRU (30,000-word vocabulary, 50-dimensional embedding layer) that routes inputs into five legal categories.

Abstract

The study centers on a question that treats raw performance as insufficient: when the model predicts correctly, is it looking where a jurist looks? Six encoder-based Transformers (BERT, DistilBERT, ConvBERT, ELECTRA, mBERT, XLM-RoBERTa) were fine-tuned on a dataset of 10,670 Constitutional Court individual-application decisions for violation / no-violation classification, and a novel “Explainability Score” (ES) was introduced, quantifying the overlap between SHAP-derived model focus and the passages jurists hand-marked as decisive in the same decisions.

Findings

• ConvBERT reached an 84.25% macro F1 — the best result in the literature at publication; the previous best on multi-right decisions was 67%.
• A 479-decision test set was hand-annotated by jurists; on decisions with recurring patterns — trial-within-reasonable-time, detention duration — the models attended to the same passages as domain experts.
• Tracking checkpoints during training, ES–F1 correlations were strong for BERT (r = 0.86) and XLM-RoBERTa (r = 0.92), yet no general monotonic relationship held — high performance does not guarantee jurist-like reasoning.
• The paper recommends selecting models on both high ES and high performance rather than F1 alone, connecting this to the GDPR “right to explanation” and the EU AI Act debate.

Method details

Decision texts were restricted to the subject, procedure and case-facts sections, stripping the court's assessment and judgment so the outcome cannot simply be “read off” the text. Hyperparameter search used Optuna's Bayesian optimization instead of GridSearch. In the explainability pipeline, each token's contribution was computed with SHAP's Partition Explainer; taking the median of unique SHAP values as a threshold, the model's most influential word groups were extracted. Their overlap with jurist-marked passages was measured with BLEU and ROUGE-L, and the harmonic mean of the two defines the Explainability Score.

Abstract

The premise echoes the Turing test: if a model truly learns judges' behavioral patterns, it should struggle more on decisions where the judges themselves disagreed — i.e., those with dissenting votes. To test this, five datasets were built from 12,887 Constitutional Court individual-application decisions for reproducibility checks; 146 dissenting-opinion decisions were held constant across all test sets, and five separate BERT models were trained. Model uncertainty was measured with two independent methods: softmax output differences and descriptive statistics of SHAP values.

Findings

• Models scored on average 19% lower macro F1 on dissenting-vote decisions than on unanimous ones (63.63 vs 78.79) — where judges diverged, the model wavered too.
• Absolute softmax differences were systematically lower on misclassified and dissenting-vote decisions; the mean, maximum and standard deviation of SHAP values diverged in the same direction.
• Between-group differences were statistically significant in pairwise t-tests across all five datasets (p < 0.001).
• Dissent was proposed as a natural “difficulty label” for the prediction task: taken together, the uncertainty measures can indicate whether the model will reach the same legal conclusion as the judges.

Method details

Some earlier studies had inflated accuracy by including the court's assessment sections in their data; here, assessment and judgment texts were carefully removed. Per-decision mean SHAP values, plus group means, maxima and standard deviations, were computed with explicit formulas to compare correct/incorrect predictions and dissent/no-dissent groups. SHAP-based heatmaps visualize model focus within individual decisions; the limitations of softmax-based uncertainty (its inability to separate epistemic from aleatoric uncertainty) are discussed alongside the calibration literature.

Abstract

Judgment-prediction research in Türkiye had so far focused only on individual applications; this study is the first to frame norm-control decisions — the abstract constitutionality review of legislation — as a prediction task. From 5,346 decisions spanning 1963–2025, a multi-stage pipeline produced a leakage-free dataset of 3,722 decisions with every outcome-revealing section removed; Turkish BERT and ELECTRA models were fine-tuned for annulment-vs-rejection classification. In a second stage, the alignment between the model's textual focus and the parts of the norm the court found unconstitutional was quantitatively stratified and opened to qualitative review by jurists.

Findings

• BERT reached 77.47% accuracy and 75.37% macro F1 from pre-verdict text alone — the first demonstration that abstract constitutional review outcomes are substantially predictable.
• Qualitative analysis revealed three distinct reasoning behaviors: in high-alignment cases the model correctly focused on the legally contentious wording of the norm (e.g., “civil servant or other public official”); in low-alignment cases it learned spurious correlations over irrelevant numbers and words.
• The most critical finding emerged in medium-alignment cases: alongside correct focus, the model exploited procedural “shortcuts” such as the court's name and the application type — a systemic-bias risk that boosts F1 yet is incompatible with impartial constitutional review.
• The study offers a concrete framework for the high-risk system audits envisioned by the EU AI Act: performance metrics can mask flawed reasoning unless reported together with explainability frameworks.
• The curated dataset was released under an open license on Hugging Face.

Method details

A four-stage pipeline reduces 5,346 raw decisions to 3,722: filtering out decisions without substantive review, remand decisions, and texts under 500 words, and truncating everything from the “PRELIMINARY EXAMINATION” and “EXAMINATION ON THE MERITS” headings onward to prevent outcome leakage. Labels are read solely from decision metadata. The final set has a ~1:2 annulment/rejection ratio (imbalance ratio 1.98); Optuna ran 20 trials per architecture, with macro F1 as the primary metric due to class imbalance. For explainability, tokens exceeding the 90th-percentile SHAP threshold across the 125 annulment decisions in the test set form the model's explanation; an “extractor agent” (gemini-2.5-flash) produced a ground-truth proxy by pulling the norm phrases at the center of the court's constitutional analysis from each decision. Decisions were stratified by Jaccard similarity into low (J ≤ 0.10), medium (0.10 < J ≤ 0.30) and high (0.30 < J ≤ 0.60) alignment tiers defined together with legal experts.

Abstract

The most critical threat to LLMs' legal reliability is fabricated content in statutory citations; hallucination rates of 58–88% have been reported for US case law, yet strictly codified systems like Turkish law lacked any systematic measurement infrastructure. The study fills this gap with three contributions: a 72-scenario benchmark produced by a generator–verifier–evaluator multi-agent pipeline — a different model at each layer — under active oversight by two jurists; a 5-step decision tree adapting the hallucination taxonomy to statutory citation; and the newly introduced Norm Content Fabrication (NCF) and Weighted Hallucination Severity (WHS) metrics. 504 inferences from seven LLMs (Gemini 3 Flash, Gemini 2.5 Flash, Claude Sonnet 4.6, Llama 4 Maverick, DeepSeek-V3.2, Qwen3.5-35B, Turkish-Gemma-9b) across six legal domains were evaluated with this framework.

Findings

• Fabricating non-existent articles is exceedingly rare: only 22 among 2,419 proposed citations. The dominant errors are real-yet-misapplied citations (Type 2) and inconsistent reasoning over correct articles (Type 3, the most common at 0.04–0.19) — showing that fabrication-centric hallucination definitions fall short in the legal context.
• Even the best model (Gemini 3 Flash) topped out at 0.84 F1: general-purpose LLMs are not yet reliable against the codified structure of Turkish law.
• A concrete helpfulness–hallucination trade-off emerged: Claude Sonnet 4.6 averaged 8.51 proposed articles per scenario, reaching 0.92 recall but dropping to 0.55 precision.
• The only Turkish fine-tuned model (Turkish-Gemma-9b) performed worst, empirically showing that general language adaptation alone is insufficient for legal reasoning.
• LLM-as-a-judge evaluations reached “almost perfect” agreement with two senior jurists (κ = 0.84–0.91), exceeding the inter-expert agreement itself (κ = 0.73–0.83).

Method details

In the multi-agent pipeline, the generator layer (Gemini 3.1 Pro Preview), acting as a senior law professor, produces difficulty-graded dispute scenarios with mandatory/optional article sets in structured JSON; the verifier layer (Gemini 3 Flash Preview) checks article validity and consistency with Court of Cassation case law via web search; the evaluator layer (Claude Opus 4.6) cross-examines scenario and verification report, rejecting errors such as citing repealed statutes. Splitting generation, verification and approval across different models reduces single-model self-validation bias. Of the initial 107 scenarios, jurists directly edited 44 and wrote 25 from scratch; the final set was narrowed to 6 legal domains × 3 difficulty levels × 4 questions = 72 scenarios. The evaluation framework classifies each proposed article with a 5-step decision tree (existence → set → citation → topic → content); the fifth step separates citation errors from fabricated explanations, grounding the NCF metric. WHS merges the false-positive types into a single severity scale with jurist-agreed legal-risk weights (fabrication 1.0 · irrelevant 0.8 · redundant within the same institution 0.3). Gemini 3.1 Pro Preview — distinct from all evaluated models — served as the judge; 126 outputs were independently reviewed by two jurists via a blind evaluation interface. By domain, performance peaked on the Turkish Penal Code (mean F1 = 0.72) and the Labor Law (0.69), while the heavily cross-referenced Commercial Code showed the highest content-fabrication rate (CFR = 0.56).