piflow

Features

1. Add file-based pipeline scheduling mode;
2. Comprehensively upgrade system security;
3. Add my center.

Requirements

• JDK 1.8
• Scala 2.12.18
• Spark-3.4.0（other spark version of piflow.jar should be built with code）
• Hadoop-3.3.0（other hadoop version of piflow.jar should be with code）

config.properties

  spark.master=yarn
  spark.deploy.mode=cluster...

Features

1. Add new visualization features;
2. Add a new Python base image management module;
3. Add vector database storage components such as Chroma, Faiss, Weaviate, Pinecone, and Qdrant.

Requirements

• JDK 1.8
• Scala 2.12.18
• Spark-3.4.0（other spark version of piflow.jar should be built with code）
• Hadoop-3.3.0（other hadoop version of piflow.jar should be with code）

c...

Features

1. Add ability to parse unstructured data;
2. Supports customize Server H2 database names;
3. Customized Python components optimized;
4. Template function optimized;
5. Pipeline optimized.

Requirements

• JDK 1.8
• Scala 2.12.18
• Spark-3.4.0（other spark version of piflow.jar should be built with code）
• Hadoop-3.3.0（other hadoop version of piflow.jar should be with code...

Requirements

• JDK 1.8
• Scala 2.12.18
• Spark-3.4.0（other spark version of piflow.jar should be built with code）
• Hadoop-3.3.0（other hadoop version of piflow.jar should be with code）

config.properties

  spark.master=yarn
  spark.deploy.mode=cluster
  
  #hdfs default file system
  fs.defaultFS=hdfs://10.0.85.83:9000
  
  #yarn resourcemanager host...

Requirements

• JDK 1.8
• Scala 2.12.18
• Spark-3.4.0（other spark version of piflow.jar should be built with code）
• Hadoop-3.3.0（other hadoop version of piflow.jar should be with code）

config.properties

  spark.master=yarn
  spark.deploy.mode=cluster
  
  #hdfs default file system
  fs.defaultFS=hdfs://10.0.85.83:9000
  
  #yarn resourcemanager host...