Security Testing of the Ellie.ai SaaS Platform

SaaS Platform Cybersecurity: How Our Project Began

Finnish company Ellie.ai helps large organizations understand complex data architectures and speak a common language about them. The platform brings business and IT teams together in one environment, where they can describe data structures, create business glossaries, build logical and physical models, and manage them. The SaaS platform supports Data Vault, dimensional modeling, and data mesh and integrates, in particular, with Snowflake, dbt, and Microsoft Fabric.

Why Metadata Also Needs Protection

It should be noted here that Ellie.ai primarily works with metadata, which may contain sensitive information, rather than with clients’ “raw” production data. If models and schemas reveal a company’s data structure and business logic, such information may be commercially sensitive for a bank or insurance organization.

In addition, the SaaS platform is an integrated tool. Enterprise clients interact with it through a web interface and REST API. The platform has corporate authentication mechanisms and integrates with external systems. Therefore, not only the web application required testing, but also potential risks at the intersection of components and services.

Why Ellie.ai Needed an Independent Pentest

Cybersecurity for the company is not only a technical issue. When your clients are large companies, banks, and insurance organizations, it is not enough for them to hear: “We regularly scan the system, and everything looks good.” They want independent confirmation. And for Ellie, this had a very practical significance: without an independent pentest report, passing the vendor security review could delay closing deals with corporate clients.

Therefore, Ellie.ai came to Datami with a task: conduct an independent pentest and confirm the platform’s security level. The client itself did not expect a large number of vulnerabilities, as the system was already under continuous automated scanning.

Project Task: What Needed to Be Tested

Ellie.ai’s request was quite specific: the company needed an independent security assessment of the platform and a formal pentest report that could be provided to corporate clients during the vendor security review. But before looking for vulnerabilities, it was necessary to determine exactly where we would look for them, for what purpose, and in what way.

The team’s task was to test possible attack scenarios as realistically as possible while staying within the agreed scope. Therefore, at the start, we agreed on the testing targets and format.

For this project, three separate testing areas were planned:

Thus, the task was not limited to running another scanner on top of the existing one. Datami had to independently test the web platform, AI component, and AWS environment and determine not only what weaknesses existed, but also how dangerous they were in real attack scenarios.

Preparing the SaaS Platform for Security Testing

A pentest is not a case where the team gets access and immediately starts attacking everything it sees. First, it is necessary to prepare an environment in which attack scenarios can be tested deeply enough while at the same time not creating unnecessary risk for the production system.

For this purpose, Ellie.ai created isolated test tenants so that the pentesters could actively work with the platform, test different scenarios, and exploit the weaknesses they found without interfering with the production environment.

When the environment was ready for testing, the team moved on to the three agreed areas: the web platform, AI chatbot, and AWS infrastructure.

Three Testing Areas – One View of Security

We placed the main emphasis on detailed manual testing. Automated tools were also used alongside the pentester's manual assessment to supplement the testing process and provide additional coverage.

In this project Ellie.ai’s cybersecurity could not be assessed based solely on the state of the web application. The platform operates in a cloud environment and has an AI component so the testing consisted of three separate but interconnected services: web application pentesting, AI pentesting, and an AWS security audit.

1. Web Application Penetration Testing

The web platform was tested in a grey-box format – using credentials provided by the client. This format of web application pentesting made it possible to test not only from the perspective of an external visitor, but also to explore scenarios available after authorization.

The testing was based on the OWASP Top 10 and combined manual testing with Burp Suite, sqlmap, and Nmap. The pentesters looked for web vulnerabilities and tested the possibility of their practical exploitation. And it was in the web part that we found the vulnerability whose story became central to this case study: the automated scanner had already seen it, but the actual risk level turned out to be completely different.

2. AI Penetration Testing

A separate part of the work was the pentest of the Ellie.ai Lumi chatbot. Unlike traditional testing, here it was necessary to take into account risks related to the behavior of the AI component.

One of the main areas of this AI penetration testing was prompt injection. We tested the AI chatbot for prompt injection scenarios and other risks associated with possible deviations from the intended operating logic. The testing results were included in the general recommendations for protecting the platform.

3. AWS Security Audit

Another level of testing was the AWS infrastructure, for which a white-box audit was conducted. The Datami team examined how securely the cloud environment on which the SaaS platform operates was configured.

To analyze the AWS infrastructure, we used Prowler and ScoutSuite as well as manual configuration checks: we checked for insecure settings and other weaknesses that could create additional risks for the platform.

Security Testing Results: What We Found

The project lasted a month. During this time the client promptly fixed the identified issues. This allowed the client to address the findings without waiting for the final report.

Meanwhile the platform itself also changed: Ellie released new functionality after the initial testing. Therefore Datami additionally tested the new changes so that the final assessment reflected the current state of the platform.

Despite continuous automated scanning, the independent testing identified vulnerabilities of varying severity levels. Among them were issues in the web application and risks related to the configuration of the cloud environment.

But the most interesting finding was not a vulnerability that was new to the system: the automated scanner already knew about it, and classified it as Low.

How Low Turned into Critical

It was an SSRF (Server-Side Request Forgery) vulnerability – a vulnerability in which an attacker can force the server to send requests to resources that the attacker cannot access directly.

The mere presence of SSRF did not yet explain how dangerous it was specifically for Ellie.ai. The automated scanner detected the issue, but rated the finding as Low.

During manual testing, Datami pentesters went further. The attack scenario was unusual and difficult to execute, but the team confirmed a scenario in which SSRF could lead to a critical impact on the security of the cloud environment. Practical testing showed that the automated assessment did not take into account the full scale of the risk.

In other words, the scanner did not miss the vulnerability; it underestimated it. Automation saw the weakness but was unable to follow the entire path of a potential attack. The manual pentest showed where this path could lead in a specific environment.

From Vulnerabilities to Confirmed Security

Simply identifying a vulnerability does not prevent attackers from exploiting it – so what happens next is important. Ellie promptly addressed the agreed findings while Datami verified the changes within the project.

Testing New Functionality

The platform itself also changed during the project. After the initial testing, the Ellie.ai team released new functionality, so Datami additionally tested the new changes. This way, the final assessment reflected the current state of the SaaS platform, rather than only its version at the time the work began.

Final Security Level

Within the agreed scope, the team confirmed that the priority findings had been fixed. This helped the client reduce the identified risks and prepare materials for the vendor security review. The project was completed within the agreed timeframe – approximately one month.

Recommendations for Long-Term Protection

Cloud platform security is a continuous process, and to maintain the achieved level of security as the platform develops and its infrastructure changes, the Datami team recommended that Ellie.ai move to regular testing cycles: combine continuous automated scanning with periodic pentesting and audits after significant updates.

More Than a Formal Report

At the start of the project, the pentest report was primarily a requirement from corporate clients for passing the vendor security review. However, the testing showed that this was far from a formality: Datami identified a number of vulnerabilities. So, along with the required report, Ellie received a much more important result – the opportunity to eliminate real risks that remained in the system despite continuous automated scanning.

Conclusion

We thank the Ellie.ai team for their trust, the opportunity to enrich our experience in assessing SaaS platforms, and the chance to demonstrate Datami’s expertise in practice.

For Ellie.ai, this project resulted in specific improvements to the platform’s security and its ability to work with enterprise clients:

• A Critical-risk SSRF vulnerability, rated as Low by the scanner, was confirmed.
• XSS and insecure AWS configurations were identified.
• The platform’s security level was improved to good.
• New functionality was tested after its release.
• A pentest report was prepared for vendor security review.
• The security barrier to enterprise deals was removed.

The client’s automated scanner did not fail at its task – it found the vulnerability. The problem was different: it was unable to go further and correctly determine its criticality for the system.

The case showed that automated scanning and manual pentesting perform different roles: the former helps quickly identify potential issues, while the latter assesses their real impact in a specific environment.

The pentest, which at the start of the project was primarily needed to pass corporate client security reviews, ultimately helped identify a real issue and address potential attack scenarios.

You can read how the Ellie.ai team itself evaluated this Datami project in the review on Clutch.