Data science has become a field where execution speed matters as much as analytical quality. Data scientists spend a significant part of their time on recurring tasks: cleaning data, writing SQL queries, choosing the right statistical test, or formatting results for an executive audience. With the rise of large language models, many teams have started to delegate these tasks to conversational agents. The promise is appealing: less manual work, more time for deep analysis, and sustained productivity.
However, turning that promise into reality is not so simple. One of the most popular strategies is to equip the model with reusable skills, often called skills: files containing instructions, examples, and criteria for a family of tasks. The idea is to avoid starting from scratch on every request while embedding expert knowledge. When these skills are written by human experts, they can reflect years of best practices. But writing and maintaining them for every analytical domain creates a significant bottleneck. That is why the possibility of having the model itself generate these skills automatically has gained interest.
A recent experiment examined whether LLM-generated skills actually improve the performance of data scientists. The study covers several stages of an analytical project lifecycle: data preparation, information extraction, statistical analysis, and report generation. For each stage, one generated skill was created and compared with a base strategy of sending only the task description to the model. The results do not show a reliable improvement: in practice, the model obtains similar outcomes with the full skill and with no skill.
This finding invites us to reflect on how we actually use generative models. Writing an instruction document and injecting it into the prompt is not enough. In many cases, additional context adds noise before clarity. The study even observes that a skill filled with irrelevant content can behave similarly to a well-intentioned skill. This suggests that format, structure, or the simple presence of metadata matters more than we think, and that the specific content of the skill does not always provide a measurable advantage.
For a technology company, these results have significant strategic value. At Q2BSTUDIO we work every day with organizations that want to integrate artificial intelligence into their data processes. The temptation to create a large repository of LLM-generated skills and rely on them as a universal solution is understandable, but the evidence suggests that a more rigorous approach is needed. AI should serve a well-designed workflow, not replace the design of the workflow.
Part of the problem lies in the nature of data tasks. An SQL query may seem simple, but its quality depends on the database schema, business semantics, and the expectations of the end user. A statistical test is not a fixed recipe: it requires knowing the data distribution, model assumptions, and decision context. A generated skill can capture a general view, but it will hardly replace the judgment an analyst needs to adapt it to a specific problem.
This does not mean that generated skills are useless. It means they should be treated as working hypotheses, not conclusions. A mature AI organization evaluates every component before putting it into production. It builds validation sets, measures performance by task type, and decides when to use a skill, a simple prompt, or an automated workflow. That experimentation culture is exactly what separates an interesting pilot from a real transformation.
In this context, custom software plays an essential role. An enterprise data platform is not a simple chat with a language model; it is an environment where data pipelines, quality controls, permissions, audits, and dashboards coexist. Q2BSTUDIO helps companies build custom software applications that integrate AI agents in a secure and controlled way. Instead of relying exclusively on a generic prompt, these applications can orchestrate models, verify results, and give data scientists a reliable starting point.
The cloud is also part of the answer. Models need access to current data, managed identities, and scalable environments. An architecture based on AI solutions on AWS/Azure cloud can connect models to systems of record without exposing sensitive information. Cybersecurity thus becomes an enabler: we cannot trust the results of an LLM if we do not know who accesses what, how decisions are logged, and what defense mechanisms exist against prompt injection attacks or data leaks.
Moreover, data analysis does not end with a statistical result. In most companies, information is consumed through dashboards, executive reports, and alerts. BI/Power BI tools make it possible to visualize results and share them with business teams. An LLM-generated skill could help prepare transformation code, but data governance and report design require a software layer that ensures consistency and traceability. At Q2BSTUDIO we integrate BI/Power BI with AI processes so that data reaches decisions in a clear and auditable way.
Another important lesson is the need to separate signal from noise. In the experiments reviewed, performance with the full skill was similar to performance without a skill, and also similar to irrelevant text with the same format. This indicates that, in many cases, the model already knows enough about the task when it is presented explicitly. Adding more documentation can be counterproductive: it consumes tokens, introduces ambiguity, and redirects the model's attention to details that are not relevant for the specific sample.
The solution is not to eliminate skills, but to redesign their use. Instead of injecting a long document into every conversation, we can build a retrieval system that selects only the part relevant to the user's question. We can also design the skill as a short checklist, not as a treatise. And we can assign AI specific tasks within a broader software process, where each step is validated by a business rule or by a human when risk justifies it.
AI agents, understood as systems capable of planning and executing actions, benefit from the same logic. An agent that cleans data should not act blindly; it should consult the real schema, check the consistency of values, document changes, and request approval when it detects an anomaly. Q2BSTUDIO designs this type of agent on the basis of robust software, integrating language models as one more component, not as the center of the universe.
In summary, the initial question, whether LLM-generated skills make data scientists better, has a nuanced answer. They can help in certain contexts, but they are not an automatic improvement. Empirical studies show that, as a default strategy, adding a generated skill does not reliably outperform a simple prompt. Organizations that will obtain the most value will be those that integrate generative AI into a data and software engineering framework, with continuous measurement, security, and focus on business outcomes.
At Q2BSTUDIO we believe in technology designed for real operations. That is why we combine the power of language models with custom software, AWS/Azure cloud architecture, cybersecurity, and BI/Power BI platforms. The goal is not to have the perfect skill, but to build systems that help people make better decisions. AI is an extraordinary tool, but it needs context, control, and judgment. That judgment, fortunately, remains human.
