ploosh.

Documentation

This connector is used to read JSON files using Spark.

:warning: A spark connector can be use only with another spark connector. It is not possible to use a spark connector with a non spark connector.

See Spark documentation for more information.

Connection configuration

No connection is required by this connector

Configuration

Test case configuration

NameMandatoryDefaultDescription
pathyesPath to the JSON file
multilinenoTrueHandles multi-line JSON files
encodingno"UTF-8"Encoding to use for UTF when reading/writing
lineSepno"\n"Character used to denote a line break

Example

Example JSON Spark:
  source:
    type: json_spark
    path: data/employees/example.JSON
    multiline: Tre
    encoding: "UTF-8"
    lineSep: "\r\n"
  expected:
    type: sql_spark
    query: |
      select * 
          from employees
          where hire_date < "2000-01-01"