Community

This is the official repository for community contributed modules. Help us grow this new ecosystem by sharing your scrapers!

How to contribute?

1. Fork this repository
2. Create a new folder in modules (e.g, modules/linkedin)
3. Add your mod.kdl file (comments should be added using DocBlock formatting)
4. Add a brief README.md in your module.
5. Open a pull request.

Standard Format

We want to maintain consistency and document, each module should also contain a doc header and each action/evaluator should have its own documentation for its behavior, slots and parameters.

Example

/**
 * @module my_module
 * @targets example.com
 * @description It does something.
 * @example
 * example.do_something {
 *   text { text }
 *   id { attr "id" }
 *   link { my_module.get_result_link }
 * }
 */
 module my_module {
  /**
   * @action do_something
   * @param {string} text - The search text
   * @slot Expects fields to define what data to pull from each search result.
   * @desc Inputs a search box and extracts results
   */
  action do_something {
    $ "#search" {
      type "=text"
    }
    $ "#submit" {
      click
    }
    wait_until
    $$ ".some-class" {
      extract "results" {
        slot
      }
    }
  }

  /**
   * @evaluator get_result_link
   * @returns {string} - The link from the result.
   */
  evaluator get_result_link {
    $ ".link"
    attr "href"
  }
}

Modules

Name	Description	Target Site
redfin	Search and extraction logic for listings.	redfin.com
[Add Your]	Submit a PR to add yours!