Skip to content

Kafka2Hdfs问题 #1

@soufunlab

Description

@soufunlab
  rdd.foreachPartition {
    partitionOfRecords =>
      val connection = HdfsConnection.getHdfsConnection(config)
      partitionOfRecords.foreach(
        record => {
          // connection.writeUTF(record)
          connection.write(record.getBytes("UTF-8"))
          connection.writeBytes("\n")
        }
      )
      // 每次完了之后进行 flush
      try {
        connection.hflush()
      } catch {
        case e: Exception => logger.error(s"hflush exception: ${e.getMessage}")
      }
  }

多个partition往一个hdfs路径写数据不会报错吗?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions