ON-PREMISE OPEN-WEIGHT LARGE LANGUAGE MODELS FOR INTERPRETABLE WEB VULNERABILITY DETECTION: A BENCHMARK AGAINST CLOUD MODELS AND RULE-BASED STATIC ANALYSIS

2026

Closed access Metadata (CSL-JSON)
DOI 10.66571/tsarka-3134-6057-11
Data confidence 70 medium
How this score is calculated

It is built from checkable properties of the record itself: whether the article has a DOI, how many independent sources confirmed its data, whether the required fields are filled in, and how recent the latest confirmation is. Disagreement between sources and a missing identifier lower it. A value of 80 or above means the record was published without further checking; 60 to 79 means it is published at reduced confidence, with a manual check advisable. A record verified by an editor by hand is never pushed below 90 by a recalculation. This scores our data about the article, not the article itself.

Abstract

Organisations handling regulated or proprietary source code are frequently prohibited from transmitting it to third-party inference providers, which excludes commercial cloud large language models from their security-analysis pipelines regardless of the accuracy those models achieve. Whether self-hosted open-weight models can substitute for them on ordinary developer hardware has not been established. This study benchmarks five quantised open-weight models, from 1.5 to 8 billion parameters and executed entirely on a consumer laptop graphics processor with eight gigabytes of video memory, on 120 annotated web-application code snippets spanning five weakness categories. Each was evaluated under zero-shot, few-shot and chain-of-thought prompting and compared against a rule-based static analyser and previously published cloud-model results on the identical corpus. The strongest local configuration reached an F1-score of 0.900, exceeding the previous-generation cloud flagship and trailing the current one by 4.7 points, while generating a remediation suggestion for every correct detection. The analysis further shows that F1 alone conceals degenerate behaviour: several configurations, including one cloud model, approach the score of a classifier labelling every input as vulnerable, so reporting specificity alongside F1 is proposed as a minimum requirement. Privacy-preserving on-premise vulnerability analysis is thus attainable without specialised infrastructure.