Objects Comparator

:revnumber: v1.1
:revdate: 15.03.2022
:revremark: initial release
:author: Amadou Mamane
:email: mamanesarki@yahoo.fr

Table of Contents

• 1. Project documentation
  • 1.1. Project description
  • 1.2. Examples

1. Project documentation

1.1. Project description

Objects comparator lets comparing :

• two instances of scala cases classes or

• two spark-scala dataframes

It is possible to ignore some fields of the case objects or some columns of the dataframes.

This could be used to compare dataframes before writing into a Cassandra table. Hence it reduces the tombstones.

1.2. Examples

1.2.1. Case objects comparison

  case class Student(id: Int, firstName: String, lastName: String, age: Int, father: String, teachers: Seq[String])

  val A = Student(1, "John", "Doe", 12, "Bob", Seq("Elisabeth", "Paul"))
  val B = Student(1, "John", "Doe", 12, "Bob", Seq("Elisabeth", "Paul"))
  val C = Student(1, "John", "Doe", 12, "Bob", Seq("Elisabeth", "Paul", "Robert"))

  import CompareIgnoringFields._

  scala> A.compareIgnoringFields(B)(IgnoredFields('teachers), CompareIgnoringFields[Student])
  res29: Boolean = true


  scala> A.compareIgnoringFields(C)(IgnoredFields('teachers), CompareIgnoringFields[Student])
  res31: Boolean = true


  scala> A.compareIgnoringFields(C)
  res33: Boolean = false

  scala> A.compareIgnoringFields(C)(IgnoredFields(), CompareIgnoringFields[Student])
  res34: Boolean = false

1.2.2. Dataframes comparisons

  val dfA = Seq((1, "John", "Doe", 12, "Bob", Seq("Elisabeth", "Paul")),
    (2, "Claire", "Michel", 13, "Bill", Seq("Elisabeth", "Paul")),
    (3, "Michael", "Marie", 12, "Johny", Seq("Elisabeth", "Paul")),
    (4, "Alice", "Doe", 12, "Rachel", Seq("Elisabeth", "Paul", "Edy"))).toDF("id", "firstName", "lastName", "age", "father", "teachers").as[Student]

  val dfB = Seq((1, "John", "Doe", 12, "Bob", Seq("Elisabeth", "Paul")),
    (2, "Claire", "Michel", 15, "Bill", Seq("Elisabeth", "Paul")),
    (3, "Michael", "Marie", 12, "Johny", Seq("Elisabeth", "Paul", "Rachel")),
    (5, "Alice", "Doe", 12, "Rachel", Seq("Elisabeth", "Paul", "Edy"))).toDF("id", "firstName", "lastName", "age", "father", "teachers").as[Student]

  scala> dfA.show(false)
  +---+---------+--------+---+------+----------------------+
  |id |firstName|lastName|age|father|teachers              |
  +---+---------+--------+---+------+----------------------+
  |1  |John     |Doe     |12 |Bob   |[Elisabeth, Paul]     |
  |2  |Claire   |Michel  |13 |Bill  |[Elisabeth, Paul]     |
  |3  |Michael  |Marie   |12 |Johny |[Elisabeth, Paul]     |
  |4  |Alice    |Doe     |12 |Rachel|[Elisabeth, Paul, Edy]|
  +---+---------+--------+---+------+----------------------+

  scala> dfB.show(false)
  +---+---------+--------+---+------+-------------------------+
  |id |firstName|lastName|age|father|teachers                 |
  +---+---------+--------+---+------+-------------------------+
  |1  |John     |Doe     |12 |Bob   |[Elisabeth, Paul]        |
  |2  |Claire   |Michel  |15 |Bill  |[Elisabeth, Paul]        |
  |3  |Michael  |Marie   |12 |Johny |[Elisabeth, Paul, Rachel]|
  |5  |Alice    |Doe     |12 |Rachel|[Elisabeth, Paul, Edy]   |
  +---+---------+--------+---+------+-------------------------+


  import CompareIgnoringFields._
  val differential = dfA.compareIgnoringFields(dfB)(Set("id"))(IgnoredFields('teachers), CompareIgnoringFields[Student])

  scala> differential("RowsWithSomeDifferencies").show(false)
  +---+---------+--------+---+------+-----------------+
  | id|firstName|lastName|age|father|         teachers|
  +---+---------+--------+---+------+-----------------+
  |  2|   Claire|  Michel| 15|  Bill|[Elisabeth, Paul]|
  +---+---------+--------+---+------+-----------------+


  scala> differential("IdenticalRows").show(false)
  +---+---------+--------+---+------+-------------------------+
  |id |firstName|lastName|age|father|teachers                 |
  +---+---------+--------+---+------+-------------------------+
  |1  |John     |Doe     |12 |Bob   |[Elisabeth, Paul]        |
  |3  |Michael  |Marie   |12 |Johny |[Elisabeth, Paul, Rachel]|
  +---+---------+--------+---+------+-------------------------+


  scala> differential("RowsNotInRight").show(false)
  +---+---------+--------+---+------+----------------------+
  |id |firstName|lastName|age|father|teachers              |
  +---+---------+--------+---+------+----------------------+
  |4  |Alice    |Doe     |12 |Rachel|[Elisabeth, Paul, Edy]|
  +---+---------+--------+---+------+----------------------+


  scala> differential("RowsNotInLeft").show(false)
  +---+---------+--------+---+------+----------------------+
  |id |firstName|lastName|age|father|teachers              |
  +---+---------+--------+---+------+----------------------+
  |5  |Alice    |Doe     |12 |Rachel|[Elisabeth, Paul, Edy]|
  +---+---------+--------+---+------+----------------------+