📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva project built a large-scale European sovereign LLM from scratch, but its low performance on Italian academic tests highlights the limits of current investment levels. This raises questions about the necessary scale for effective country-specific language models.
Italy’s Minerva-3B, a sovereign language model trained from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored just 4.9% on the INVALSI Italian school-exam benchmark, a result that questions the effectiveness of current investment levels in country-specific large language models.
The Minerva project, led by Sapienza University of Rome and supported by Italy’s national research infrastructure, built a family of models ranging from 350 million to 7 billion parameters, with weights and training data openly published from inception. Despite its scale, the 3B parameter model’s performance on the INVALSI tests—an important measure of Italian academic language proficiency—was near chance, at only 4.9%. This empirical result suggests that the current scale of investment, even with half of the training data in Italian, may be insufficient for complex language understanding tasks.
Minerva’s development involved significant institutional backing, including the CINECA supercomputing consortium and funding through Italy’s PNRR national AI strategy. The team of 15 researchers and PhD students conducted extensive training and evaluation, aiming to create a model that reflects Italian language and knowledge. However, the low test scores reveal a structural challenge: larger datasets and more parameters might be necessary to achieve meaningful country-specific language comprehension and knowledge depth.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Building A large language model with Ai: A Practical Guide to Structuring LLM Systems from Scratch Using Reverse-Engineering Techniques
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
AI Systems Performance Engineering: Optimizing Model Training and Inference Workloads with GPUs, CUDA, and PyTorch
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
supercomputing servers for AI research
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
AI model training data storage solutions
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-Language Models
The low performance of Minerva-3B on Italian academic assessments highlights a critical issue for Europe’s sovereign-language AI initiatives: scale matters. Despite significant investment and infrastructure, the empirical evidence indicates that current models may not be large enough or trained on sufficient native-language data to handle complex language tasks effectively. This challenges the assumption that moderate-scale models can fulfill national language and knowledge needs without substantial scaling, potentially affecting future policy and funding decisions across Europe.
European Sovereign LLM Strategies and Challenges
The Minerva project is part of a broader European effort to develop sovereign language models, with contrasting approaches such as Portugal’s AMÁLIA, which layered language specialization onto a multilingual foundation. While Italy’s approach involved training from scratch with a large dataset, results suggest that scale and data volume are critical for success. Previous initiatives like AMÁLIA have not yet published their weights, but the Minerva results provide a concrete benchmark, emphasizing that investment levels must be reconsidered to meet language-specific performance goals.
“The low INVALSI scores of Minerva-3B reveal a structural challenge: investing in native-language data and scale is crucial for meaningful AI in national languages.”
— Thorsten Meyer, AI researcher
Unresolved Questions About Model Scaling and Performance
It remains unclear what specific scale or data volume is necessary for models like Minerva to perform effectively on complex language tasks. The results from Minerva-3B suggest current levels are insufficient, but the exact thresholds for meaningful performance are still under investigation. Additionally, how future iterations or larger models will perform remains uncertain, as does how these findings will influence European AI policy.
Next Steps in Sovereign LLM Development and Evaluation
The Minerva team is continuing to refine their models and methodology, with upcoming evaluations planned for larger parameter scales and more extensive native-language datasets. Policymakers and researchers will likely reassess funding and infrastructure strategies in light of these findings, emphasizing the need for larger-scale investments to meet national language and knowledge requirements. Further empirical results will inform whether scaling up can overcome current limitations.
Key Questions
Why did Minerva-3B perform so poorly on Italian academic tests?
Despite training on a large dataset with 50% Italian content, the model’s size (3 billion parameters) appears insufficient for complex language understanding, highlighting the importance of scale and data volume for such tasks.
Does this mean smaller models can’t be effective for national languages?
Current evidence suggests that smaller models, even with substantial native data, may struggle with complex tasks. Larger models with more parameters and data are likely necessary for effective performance.
What implications does this have for European AI policy?
It indicates that European sovereign-Language AI initiatives should consider significantly increasing their investment in model size and native-language data to achieve meaningful results.
Is the low performance specific to Italian or generalizable?
While specific to Italian in this context, the findings imply that similar challenges may arise for other languages unless comparable scale and data investments are made.
Will future models improve performance?
Likely, if the models are scaled up and trained on larger, more comprehensive native-language datasets. Ongoing research aims to determine the exact scale needed.
Source: ThorstenMeyerAI.com
