📊 Full opportunity report: Engineering Is Automated. Research Is the Residual. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Recent assessments indicate AI systems now automate most core engineering tasks in AI research. However, the automation of AI research itself remains limited, raising questions about future progress and innovation.
Recent analysis indicates that AI systems have achieved near-complete automation of core engineering tasks in AI research, while the automation of research activities themselves remains limited, according to Thorsten Meyer and recent empirical data.
Six key benchmarks measuring AI capabilities in research-relevant tasks show rapid progress toward saturation, with some reaching near-complete automation. For example, the CORE-Bench, which assesses research reproduction, improved from 21.5% in September 2024 to 95.5% in December 2025, with the benchmark’s author declaring it ‘solved.’ Similarly, the MLE-Bench, evaluating performance on Kaggle competitions, advanced from 16.9% in October 2024 to 64.4% in February 2026, approaching mid-tier human performance.
These benchmarks indicate that AI can now handle much of the engineering work involved in reproducing research and performing competitive ML tasks. Conversely, the capacity to automate the creative and exploratory aspects of research—such as hypothesis generation and novel problem framing—remains less certain. Thorsten Meyer notes that while engineering tasks are increasingly automated, the residual research component may be fundamentally different, possibly requiring distinct approaches or remaining less automatable.
Engineering is automated.
Research is the residual.
Six skill benchmarks. Edison’s framing. The question Clark leaves open is whether research is just engineering at scale.
Jack Clark’s Import AI #455 catalogs six benchmarks measuring AI capability on AI R&D tasks and concludes “AI can today automate vast swatches, perhaps the entirety, of AI engineering.” The residual question is research. The structural read on the residual: it may not be a permanent moat.
Six skills. One trajectory.
Clark catalogs six benchmarks measuring AI capability on AI R&D-relevant tasks. Each individual benchmark could be noise. Six benchmarks moving together is a curve. The pattern is the cascade observed across the broader Clark series — visible here in the specific R&D-skill domain.
AI Tools for Finance and Accounting Professionals: Automate Tasks, Save Hours, Work Smarter
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three data points. Mixed signal.
Clark provides three data points on the creative-spark question. Yes-evidence: Erdős-1051, centaur math discovery, sporadic Move-37-style moments. No-evidence: low yield, framing dependence, absence of acceleration. The mixed signal is the honest read.
The data supports two readings. Pessimistic: rare moments suggest creative insight is qualitatively distinct from engineering work. Optimistic: rare moments are an artifact of low-volume exploration; more shots on goal yields more discoveries. Both readings are consistent with Clark’s “vast swatches, perhaps the entirety” claim. They differ on the residual.
Machine Learning with PyTorch and Scikit-Learn: Develop machine learning and deep learning models with Python
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five dimensions Clark gestures at but leaves underdeveloped.
Clark’s section is rigorous on the empirical evidence. Five strategic dimensions matter for the institutional response that the Clark series synthesis argues is structurally inadequate.
Computational Visual Media: 13th International Conference, CVM 2025, Hong Kong SAR, China, April 19–21, 2025, Proceedings, Part II (Lecture Notes in Computer Science)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Two readings. Different equilibria.
The structural question Clark leaves open: is research a permanent moat that bounds automated AI R&D, or is it engineering at scale that dissolves with more shots on goal? Both readings are consistent with the current data. They differ by orders of magnitude in consequences.
Productivity multiplier years
Recursive loop operational
Innovation in Music: Current Research Perspectives (Perspectives on Music Production)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five audiences. Asymmetric cost of being wrong.
The institutional response should not bet on inspiration being a permanent moat. If the distinction holds, capacity built is still useful. If it closes, capacity is necessary. Asymmetric cost-of-being-wrong points toward building now.
IN INDUSTRY
IN ACADEMIA
POLICYMAKERS
INVESTORS
EVERYONE ELSE
Engineering is automated. The residual is the question. The institutional response should not bet on inspiration being a permanent moat.
Implications of AI Automating Engineering Tasks
This shift suggests that the bottleneck in AI development may no longer be engineering or replication but rather the creative and hypothesis-driven aspects of research. As engineering becomes automated, the pace of AI innovation could accelerate, but questions remain about whether AI can fully automate the scientific discovery process. This has profound implications for how research teams, institutions, and industries approach AI development and innovation strategies.
Recent Progress in AI Research Automation
Over the past 18 months, multiple independent benchmarks have shown rapid advances in AI’s ability to perform core research tasks. The CORE-Bench, testing research reproduction, and MLE-Bench, evaluating Kaggle competition performance, both nearing saturation levels. These developments follow a broader pattern of AI capabilities becoming production-ready in tasks like kernel design and infrastructure optimization, signaling a transition from experimental to operational AI systems in research contexts.
Thorsten Meyer highlights that the progress across these benchmarks supports the view that AI is automating much of the engineering work involved in research, with some experts declaring certain tasks ‘solved.’ However, the residual challenge of creative research remains less well-defined and more uncertain.
“The pattern across multiple benchmarks indicates that AI is automating vast swaths of engineering tasks in AI research, and the residual—the creative research—may be less automatable than some assume.”
— Thorsten Meyer
Uncertainties Around Automating Scientific Discovery
It remains unclear how much of the research process—such as hypothesis generation, experimental design, and theoretical innovation—can be automated. While engineering tasks are nearing full automation, the residual research component may involve aspects that are inherently less automatable or require different AI capabilities. The structural question posed by Clark and Meyer about whether research itself is a form of engineering at scale is still open and under active discussion.
Next Steps in AI Research Automation and Exploration
In the coming months, further benchmarking and empirical studies will clarify the limits of AI automation in research. Industry and academia are likely to focus on developing AI systems capable of more creative and hypothesis-driven tasks, while also refining existing engineering automation. Monitoring progress in these areas will be key to understanding whether the residual research challenge can be effectively addressed within the next 32 months.
Key Questions
What are the main benchmarks indicating AI automation progress?
The CORE-Bench for research reproduction and MLE-Bench for Kaggle competition performance are two key benchmarks showing rapid progress towards automation in core research tasks.
Why is automating engineering tasks easier than research?
Engineering tasks are often well-defined, procedural, and rule-based, making them more amenable to automation. In contrast, research involves creative, hypothesis-driven work that is less predictable and harder to automate fully.
What does this mean for future AI development?
As engineering automation advances, the primary remaining challenge is automating the creative and exploratory aspects of research, which could accelerate overall AI progress but also require new approaches.
Are there risks to relying on AI for research automation?
Potential risks include overestimating AI capabilities in creative tasks, loss of human oversight, and the challenge of ensuring AI-generated research is valid and reliable. These issues need careful management as automation progresses.
Source: ThorstenMeyerAI.com
