Reflect progress percentage of native module (c++ to node.js)

[AtiqGauri]

Issue:

Data analyzation, pattern statistics and statistics import process can take hours if we supply enough large input data-set. So, we need a way to show progress percentage to user.

• Main problem is we can't continuously communicate with native module to get progress status
• Right now we get completion status once whole process completes.

Solution

Create a shared file in "Temp" folder. This will be shared by both c++ and node.js code, with this native module can update that file periodically and then node.js can read it periodically.

1. JSON object file suits well for this case, it will be well formatted and easy to understand.
   • c++ don't have any support for JSON parsing in standard library.
2. Alternatively we can use plain text file, native module can write "predefined data keywords" which then will be read by node.js to update progress based on that.

Required Progress variables

1. Process: anlayzing raw data

• - this process starts with resizing files, so we need 2 variables:
  a) total files before resizing
  b) total resized files
• - resized file then goes to analyzation process, we need 2 variables:
  a) total files before analyzation
  b) total analyzed files

2. Process: pattern statistics generation
this will read all output files generated by analyzation process and then write all identical pattern data into a unique file belongs to that particular pattern. So, we can measure progress based on how many files has been completed with pattern categorization.

• - categories patterns into their own unique files, so we need 2 variables
  a) total files generated by analyzation process
  b) total file done with pattern

Resources

1. Benchmarks for JSON library for c++
2. JSON libraries for c++
3. cereal JSON library
4. Blog on JSON parsing