By Subhaditya Mukherjee, University of Eindhoven.
If you enjoyed the read, feel free to send us a message here, or send me an email if you want to have a chat. After all, this is part of an EU project, and collaboration is always welcome.
The ALFIE AutoML Engine was developed as part of the ALFIE project, a European initiative focused on making AI more accessible while accounting for its legal, ethical, and explainability requirements. The goal is not simply to help users train machine learning models, but to make it easier for them to explore ideas, evaluate potential applications, and understand the implications of their choices along the way.
This blog post goes into detail on how we are working on building such a system and some of the design challenges we tackled over the past year. This is not meant to be a very technical blog, although hopefully you, dear reader, know most of the terms that are mentioned.
Our target users are people who want to build an AI model without necessarily having a background in machine learning. Perhaps they might want to test whether a model could work for their organization, explore an idea quickly, or determine whether a particular application is worth pursuing further. We therefore cannot assume that they know how to prepare a dataset, select an appropriate model, or evaluate the broader consequences of deploying one.
Consider a user working with a facial recognition dataset. They might recognize that their data needs to be balanced, but they may not know how to identify class imbalance, understand its consequences, or decide how to address it. They may also be unaware of legal or ethical concerns associated with the intended application. These are not necessarily things that users will think about when their immediate goal is simply to get a model working.
The engine is designed to address this gap by bringing several capabilities together in one platform. In addition to training models, it provides tools for data analysis, bias mitigation, explainable AI (XAI), and concept drift analysis. We are also working to connect users with relevant information from expert discussions through a semantic knowledge graph. An LLM-based agent ties these components together, allowing users to ask questions about their data, the training process, and the resulting reports. (Note that some of these are not strictly part of the AutoML engine but are complementary packages.)
At present, the engine supports tabular data, vision, audio, text, and combinations of these modalities. We have also introduced a category called AutoML+ for pipelines that combine large language models (LLMs) or vision-language models with conventional tools. The project includes several use cases, including a vision task, a multimodal text-and-vision task, and a website accessibility pipeline.
This post will focus on how the engine works, the architectural decisions behind it, and some of the challenges involved in building a system for users who may not have extensive technical knowledge.
From raw data to a usable model
The first challenge is getting from a user’s data to something that can be used for training. This is handled by a separate data-processing component rather than by the AutoML Engine itself, but it is an important part of the overall workflow.
The process begins by identifying the intended task and loading the data in an appropriate format. For example, a user might want to perform a tabular classification task. Once the data has been loaded, the system performs preprocessing and generates an initial report describing its characteristics and potential problems. This report might highlight class imbalance, missing values, or columns that could be more or less relevant to the task. The purpose is to give users an initial understanding of their dataset before training begins, particularly when they might not otherwise know what to look for. We also attempt to mitigate certain issues automatically. If the dataset contains class imbalance, for instance, the system may use oversampling or undersampling to address it. The user is informed about what is happening and can use the accompanying explanations to make a decision about how to proceed.
This is where the LLM agent becomes particularly useful. Rather than expecting users to understand every report or technical term, we provide an interface through which they can ask questions about the outputs. The agent is connected throughout the platform, so it can help users interpret information produced by the different components.
The intention is to make the process more accessible without hiding every decision behind automation. Users should be able to identify potential problems, understand the actions taken by the system, and decide whether they are comfortable proceeding.
Making data requirements understandable
Designing the data input process is more complicated than it might initially seem. If we were building a tool exclusively for AI researchers or computer scientists, we could specify the required format and expect users to prepare their datasets accordingly. In practice, someone familiar with machine learning might write a custom data loader, transform files with a script, or adapt an existing dataset to suit a particular pipeline. We cannot make the same assumptions about our target users since they might simply have a collection of data and an idea they want to test.
Consider a vision classification task. We require a spreadsheet containing filenames and their corresponding labels, along with a folder containing the actual images in the expected structure. These requirements are straightforward from an implementation perspective, but they introduce an additional task for someone who may not know how machine learning datasets are usually organized.
We could try to detect arbitrary folder structures automatically or use an AI agent to reorganize the data. However, these approaches introduce additional complexity and can make the process less transparent. For now, we prioritize clear requirements and a relatively simple input process over attempting to accommodate every possible data structure.
After uploading a dataset, users receive a report if the system identifies potential problems. They can then address those issues or decide whether to proceed despite them. This is particularly important when the data raises potential legal or ethical concerns that could have consequences for the eventual deployment of the model. The aim is to bring these issues to the user’s attention early, when there is still an opportunity to reconsider the data or the intended application.
Choosing the right level of automation
Once the data has been prepared, the next challenge is training a suitable model. However, before discussing the individual training pipelines, it is worth clarifying what we mean by AutoML in this project.
The engine is not a fully automated AutoML system in the strictest sense. A more comprehensive approach might involve neural architecture search, modifying model architectures, or training models from scratch. Although these methods are possible, they can require substantial computational resources and do not necessarily offer the best trade-off for our intended use cases.
In many situations, starting with a pretrained model and fine-tuning it for a particular task is a more practical approach. This is the general direction we follow throughout the project.
The distinction reflects an important design decision: we are optimizing for useful results and accessibility, rather than for maximum model performance at any cost.
Our users may want to establish whether an idea works before committing significant time and resources to it. They do not necessarily need a state-of-the-art model at the first stage of experimentation. Achieving the last few percentage points of performance can require more specialized training, additional information, and considerably more computation. Automating this process for every possible task would be expensive and would not necessarily provide enough value to justify the cost. Instead, we aim to provide a reasonably good starting point, together with enough information for users to decide whether further investment is worthwhile.
Tabular data: leveraging AutoGluon
For tabular tasks, we use AutoGluon, a library that provides a robust framework for tabular classification and other tabular machine learning problems.
At a high level, AutoGluon works with a portfolio of models and training approaches. Given a task, it evaluates suitable candidates, trains models independently and in combination, and uses techniques such as stacking to identify a strong-performing model. This is a practical fit for our use case. Tabular models are generally less computationally demanding than many vision or multimodal models, making it feasible to explore several candidates without incurring excessive training costs.
Using an existing library also allows us to benefit from an established approach to model selection rather than implementing a separate training strategy for every tabular task.
Vision, audio, text, and multimodal data: a portfolio-based approach
For vision, audio, text, and multimodal tasks, the training process becomes more complicated. The range of possible models and configurations is larger, and the cost of experimentation can increase substantially.
Rather than attempting extensive architecture search, we maintain a portfolio of candidate models (that are generally good for transfer tasks, rather than being trained for a specific task) and use Optuna (a hyperparameter search library) to perform an optimized hyperparameter search over them. We evaluate the candidates using metrics appropriate to the task, with the possibility of allowing users to provide their own metrics in the future. (Note: This functionality is not yet implemented.)
Once the search is complete, we select the best-performing model, save it to the database, and run inference on the test or validation set, depending on the task and pipeline.
The same general approach is used across several tasks, including image classification, video segmentation, object detection, audio classification, text classification, and multimodal tasks involving combinations such as images and tabular data. The details vary between tasks, but the underlying principle remains consistent: select suitable candidates, evaluate them efficiently, and return a useful model without exposing users to an unnecessarily complicated optimization process.
We also provide access to candidate models rather than presenting only the final selection. Users can make choices based on their priorities, such as preferring a smaller model, faster training, or potentially better performance at the cost of additional time. Other training options are available as well, although I will not discuss them in detail here. This flexibility matters because the model with the highest score is not automatically the most suitable choice for every application.
Beyond training: understanding the model
A trained model is only one part of the output we want to provide. Users also need ways to understand its behaviour, identify potential problems, and make informed decisions about what to do next.
Two components are particularly relevant here: the explainable AI pipeline and the connection to expert knowledge.
Explainable AI
The XAI pipeline takes a trained model and a subset of the validation data and attempts to identify the factors influencing its predictions. For example, it might indicate which parts of an image the model focuses on during classification or which words influence a text prediction.
These explanations provide another way to inspect a model before deployment. A model might achieve good evaluation scores while relying on information that is not relevant to the intended task. Examining its behaviour can help users identify unexpected patterns and investigate whether further changes are necessary.
XAI does not replace conventional evaluation metrics, nor does an explanation automatically establish that a model is reliable. Instead, it complements the training results by providing a different perspective on how the model behaves. For our target users, this additional perspective is particularly important. They may not have the expertise to design extensive diagnostic experiments themselves, so providing accessible reports can help them identify questions that deserve further investigation.
Bringing expert knowledge into the workflow
The other component we are developing connects the engine to information from expert forums.
We have established a forum where experts can comment on topics created by moderators. Participants can vote on contributions, provide their own answers, and ask questions. A semantic knowledge graph is then used to connect relevant information from these discussions to the AutoML pipeline. I will leave the implementation details of the knowledge graph for a future post.
Initially, we had hoped to use this information to influence model selection or modify specific parts of the training pipeline. However, we found that this was not feasible within our current constraints. We therefore decided to focus on making the information available to users as additional context. For example, a user might describe a task through the chat interface and upload a tabular dataset. If the task resembles something previously discussed in the forum, the agent will try to retrieve relevant information, including warnings about legal concerns or potential consequences associated with a similar application.
This approach attempts to provide users with information they might not otherwise know to look for. This is important because some problems can be addressed directly by the system, while others require the user to reconsider the application itself. In the latter case, making the relevant information available early may be more useful than attempting to automate a decision that depends on the context of the task.
AutoML+: combining language models with conventional tools
Not every use case in the ALFIE project fits neatly into a conventional model-training pipeline. This is why we introduced AutoML+, a category for workflows that combine LLM-based tools or vision-language models with other, more static tools.
A useful example is the website accessibility pipeline. Say we want to use an LLM to inspect website code and identify potential accessibility issues, such as problems with colour contrast or text sizing on mobile devices. At the same time, we also want to examine specific properties of the website independently. For example, we can check whether the alternative text associated with an image accurately describes its contents and notify the user if there appears to be a mismatch.
The pipeline then might also be required to incorporate static tools that evaluate properties such as word difficulty, the presence of long words, and how difficult the text is to understand. Furthermore, additional checks can be introduced over time. The result is a report that identifies potential changes in relation to accessibility guidelines, including specific issues that users can investigate and address.
This differs from the conventional AutoML workflow because the system is not simply training a model and returning its predictions. Instead, several tools perform different tasks, and their outputs are brought together to produce a final report.
Designing these pipelines introduces its own architectural challenges. We need to determine how the individual tools should interact, how intermediate results should be handled, and how to present the final output coherently. The result is a more involved workflow, but one that allows us to address use cases that do not fit the usual model-training paradigm.
Designing for users, not just researchers
Across these components, one of the recurring challenges has been deciding how much functionality to expose and how much complexity to introduce.
As a researcher and one of the main contributors to this project, I sometimes find it difficult to scale down my ideas. It is tempting to add more evaluation metrics, explore additional training approaches, or provide users with a large number of configuration options. However, the needs of our target users are different from those of researchers who want to investigate every aspect of a model. We need to be selective about which features we
implement and ensure that each one contributes to the user’s ability to accomplish a task.
For example, providing ten evaluation metrics is not necessarily more useful than providing two or three that are particularly relevant. Users who are unfamiliar with these metrics may struggle to interpret them, even if an LLM is available to explain the results. More information does not automatically lead to better decisions.
This consideration affects both the architecture and the user experience. We need to make sure that the components work together, but we also need to decide which information should be presented, which decisions can be automated, and where users should retain control.
The same principle applies to legal and ethical concerns. If a task raises potential compliance issues, including concerns related to GDPR, the system should bring them to the user’s attention as early as possible. If an issue can be mitigated automatically, we can attempt to address it. If it cannot, we should provide the information needed for the person responsible for the model to decide how to proceed.
The engine cannot make every decision on behalf of the user, and its reports do not guarantee that a model is legally compliant or ethically appropriate. Our aim is to make potential problems more visible and give users a better basis for deciding what to do.
Ultimately, the challenge is to strike a balance between technical capability and practical usability. We want the engine to be powerful enough to support a range of tasks, but straightforward enough that users do not need extensive machine learning expertise to get started.
Future directions
There are several areas we would like to explore as the project develops. Some involve improving the existing infrastructure, including better deployment support, faster model training, and more effective experiment tracking. Others concern the training process itself, such as supporting different combinations of multimodal models that users can fine-tune separately.
We would also like to explore more fine-grained training options. These could remain hidden from the default interface while being made available to more advanced or professional users who want greater control over the process.
For AutoML+, a key direction is to develop more unified pipelines that can accommodate a wider variety of tasks. Other possibilities include mechanisms for suggesting prompt optimizations and support for running larger-scale studies.
There is considerable room for experimentation, but we will need to evaluate these possibilities against the same criteria that guide the current design: whether they provide meaningful value to our users and whether the additional complexity is justified.
Final thoughts
The ALFIE AutoML Engine is an attempt to bring model training, data analysis, explainability, and relevant expert knowledge together in a single platform. The goal is not to automate every possible aspect of machine learning, but to make it easier for users to experiment with AI while understanding the implications of their choices.
Achieving this requires more than selecting suitable models. We also need to consider how users provide their data, how potential problems are communicated, how different tools interact, and how the resulting information can support decisions without overwhelming the user.
The balance between automation and transparency is central to the project. We want users to obtain a useful model quickly, but we also want them to recognize when the data, the model’s behaviour, or the intended application warrants further investigation. If an initial experiment is promising, they can then decide whether to invest additional time in improving the model, refining the data, or addressing the issues identified by the system.
There is still plenty of work to do, both in improving the existing pipelines and exploring new approaches. For now, the priority is to build a system that makes AI experimentation more accessible while keeping its limitations and potential consequences visible.