Small Files Compactor

:revnumber: v1.1
:revdate: 15.03.2022
:revremark: initial release
:author: Amadou Mamane
:email: mamanesarki@yahoo.fr

Table of Contents

• 1. Project documentation
  • 1.1. Project description
  • 1.2. Packages descriptions

1. Project documentation

1.1. Project description

Small files compactor helps resolving the recurring problem of small files on Hadoop clusters.

1.2. Packages descriptions

Here is a brief description for each package and object in the source code

./src/main/scala/com/mydevs/database/database.scala

/**
  * Provides database or data related classes to handle data in/out.
  *
  * ==Overview==
  * This package contains common classes to handle some common functions
  * such as read/write data from/to Hive, Cassandra or external RDBMS.
  * This package handle also any sqoop related functions (import/export).
  */
package object database

./src/main/scala/com/mydevs/database/Compactor.scala

/**
 * API for writing data into Hadoop according to HDFS block size
 *
 * ==Overview==
 * This includes
 *  - A method for writing data into hive tables by producing the exact number of files expected (data volume divided by HDFS block size)
 *  - A faster method for writing data into hive tables but it produces the approximate number of files expected
 *  - A method for writing into HDFS directory by producing the exact number of files expected
 *
 *  Note that this methods support partitioned tables
 */
object Compactor