Features

1. Add ability to parse unstructured data;
2. Supports customize Server H2 database names;
3. Customized Python components optimized;
4. Template function optimized;
5. Pipeline optimized.

Requirements

• JDK 1.8
• Scala 2.12.18
• Spark-3.4.0（other spark version of piflow.jar should be built with code）
• Hadoop-3.3.0（other hadoop version of piflow.jar should be with code）

config.properties

  spark.master=yarn
  spark.deploy.mode=cluster
  
  #hdfs default file system
  fs.defaultFS=hdfs://master:9000
  
  #yarn resourcemanager.hostname
  yarn.resourcemanager.hostname=master
  
  #if you want to use hive, set hive metastore uris
  #hive.metastore.uris=thrift://master:9083
  
  #show data in log, set 0 if you do not want to show data in logs
  data.show=5
  
  #server ip and port, ip can not be set to localhost or 127.0.0.1
  server.ip=your_ip
  server.port=8002
  
  #h2db port
  h2.port=50002
  #h2.name=test
  
  monitor.throughput=false
  #If you want to upload python stop,please set hdfs configs
  #example hdfs.cluster=hostname:hostIP
  hdfs.cluster=master:127.0.0.1
  hdfs.web.url=master:50070
  checkpoint.path=/piflow/tmp/checkpoint/
  
  #unstructured.parse
  unstructured.parse=false

Command

  ./start.sh
  ./stop.sh
  ./restart.sh
  ./status.sh