Get Started

Ploosh is open source yaml based framework used to automatize the testing process in data projects.

Get Started Learn More

What's ploosh?

Ploosh is an innovative testing framework designed to automate the validation processes in your data projects. Built around YAML configuration, Ploosh allows you to quickly compare datasets and ensure they meet expected results, all with minimal code.

Why Choose Ploosh?

  • Effortless Automation: Ploosh simplifies test automation, especially for projects handling large volumes of data. Its YAML-based structure offers clear, easy-to-manage test cases and configuration files.
  • Fast and Powerful: With its intuitive interface and streamlined commands, Ploosh drastically reduces the time required to set up and run comprehensive tests. Results can be exported in multiple formats (JSON, CSV), giving you full flexibility for analysis.
  • Seamless Integration: Whether you're working with SQL databases, file systems, or APIs, Ploosh integrates smoothly into your existing data pipelines. It can be employed in both DevOps environments and traditional software development workflows.

Key Features

  • YAML-Based Testing: Define your connections and test cases in well-structured YAML files.
  • Data Comparison: Automatically compare source datasets against expected outcomes with high precision.
  • Result Exporting: Generate detailed test reports in various formats to facilitate analysis and traceability.

Get started

Steps

  1. Install the ploosh package
  2. Setup a connection file
  3. Setup the test cases
  4. Run tests
  5. Get results

Installation

Install the ploosh package from PyPi package manager with the following command

pip install ploosh

Setup a connection file

Add a yaml file with name "connections.yml" and following content:

mssql_getstarted:
  type: mysql
  hostname: my_server_name.database.windows.net
  database: my_database_name
  username: my_user_name
  # using a parameter is highly recommended
  password: $var.my_sql_server_password

Setup the test cases

Add a folder "test_cases" with a yaml file with any name. In this example "example.yaml". Add the following content:

Test aggregated data:
  options:
    sort:
      - gender
      - domain
  source:
    connection: mysql_demo
    type: mysql
    query: |
      select gender, right(email, length(email) - position("@" in email)) as domain, count(*) as count
        from users
        group by gender, domain
  expected:
    type: csv
    path: ./data/test_target_agg.csv

Run tests

ploosh --connections "connections.yml" --cases "test_cases" --export "JSON" --p_my_sql_server_password "mypassword"

Test results

[
  {
    "name": "Test aggregated data",
    "state": "passed",
    "source": {
      "start": "2024-02-05T17:08:36Z",
      "end": "2024-02-05T17:08:36Z",
      "duration": 0.0032982
    },
    "expected": {
      "start": "2024-02-05T17:08:36Z",
      "end": "2024-02-05T17:08:36Z",
      "duration": 0.0012451
    }
  }
]

Run with spark

It's possible to run the tests with spark. To do that, you need to install the spark package or use a platform that already has it installed like Databricks or Microsoft Fabric.

See the Spark connector for more information.

Read our blog