Skip to content

remove kiba from models #62

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dgonzdev merged 4 commits into from
Feb 14, 2026
Merged

remove kiba from models #62

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
72757f7
remove kiba from models
dgonzdev Feb 14, 2026
a8f3af2
update local setup
dgonzdev Feb 14, 2026
c9eb56a
update your nyc open data
dgonzdev Feb 14, 2026
7a8d7c2
update etl
dgonzdev Feb 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 0 additions & 38 deletions app/models/department_of_transportation/bicycle_counter.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,43 +193,5 @@ def self.import_soda3_csv
end
end
private_class_method :import_soda3_csv

def self.run_import(api_version: '2', content_type: 'json')
if api_version == '2' && content_type == 'json'
run_import_soda2
end

if api_version == '2' && content_type == 'csv'
run_import_soda2_csv
end

if api_version == '3' && content_type == 'json'
run_import_soda3
end

if api_version == '3' && content_type == 'csv'
run_import_soda3_csv
end
end

def self.run_import_soda2
Etl::Runners::PrimaryDbImports::BicycleCounters::Soda2IntoPrimaryDb.run
end
private_class_method :run_import_soda2

def self.run_import_soda2_csv
Etl::Runners::PrimaryDbImports::BicycleCounters::Soda2CsvIntoPrimaryDb.run
end
private_class_method :run_import_soda2_csv

def self.run_import_soda3
Etl::Runners::PrimaryDbImports::BicycleCounters::Soda3IntoPrimaryDb.run
end
private_class_method :run_import_soda3

def self.run_import_soda3_csv
Etl::Runners::PrimaryDbImports::BicycleCounters::Soda3CsvIntoPrimaryDb.run
end
private_class_method :run_import_soda3_csv
end
end
38 changes: 0 additions & 38 deletions ...s/office_of_citywide_event_coordination_and_management/nyc_permitted_event_information.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,43 +233,5 @@ def self.import_soda3_csv
end
end
private_class_method :import_soda3_csv

def self.run_import(api_version: '2', content_type: 'json')
if api_version == '2' && content_type == 'json'
run_import_soda2
end

if api_version == '2' && content_type == 'csv'
run_import_soda2_csv
end

if api_version == '3' && content_type == 'json'
run_import_soda3
end

if api_version == '3' && content_type == 'csv'
run_import_soda3_csv
end
end

def self.run_import_soda2
Etl::Runners::PrimaryDbImports::NycPermittedEventInformation::Soda2IntoPrimaryDb.run
end
private_class_method :run_import_soda2

def self.run_import_soda2_csv
Etl::Runners::PrimaryDbImports::NycPermittedEventInformation::Soda2CsvIntoPrimaryDb.run
end
private_class_method :run_import_soda2_csv

def self.run_import_soda3
Etl::Runners::PrimaryDbImports::NycPermittedEventInformation::Soda3IntoPrimaryDb.run
end
private_class_method :run_import_soda3

def self.run_import_soda3_csv
Etl::Runners::PrimaryDbImports::NycPermittedEventInformation::Soda3CsvIntoPrimaryDb.run
end
private_class_method :run_import_soda3_csv
end
end
38 changes: 0 additions & 38 deletions ..._citywide_event_coordination_and_management/nyc_permitted_event_information_historical.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,43 +233,5 @@ def self.import_soda3_csv
end
end
private_class_method :import_soda3_csv

def self.run_import(api_version: '2', content_type: 'json')
if api_version == '2' && content_type == 'json'
run_import_soda2
end

if api_version == '2' && content_type == 'csv'
run_import_soda2_csv
end

if api_version == '3' && content_type == 'json'
run_import_soda3
end

if api_version == '3' && content_type == 'csv'
run_import_soda3_csv
end
end

def self.run_import_soda2
Etl::Runners::PrimaryDbImports::NycPermittedEventInformationHistorical::Soda2IntoPrimaryDb.run
end
private_class_method :run_import_soda2

def self.run_import_soda2_csv
Etl::Runners::PrimaryDbImports::NycPermittedEventInformationHistorical::Soda2CsvIntoPrimaryDb.run
end
private_class_method :run_import_soda2_csv

def self.run_import_soda3
Etl::Runners::PrimaryDbImports::NycPermittedEventInformationHistorical::Soda3IntoPrimaryDb.run
end
private_class_method :run_import_soda3

def self.run_import_soda3_csv
Etl::Runners::PrimaryDbImports::NycPermittedEventInformationHistorical::Soda3CsvIntoPrimaryDb.run
end
private_class_method :run_import_soda3_csv
end
end
6 changes: 6 additions & 0 deletions docs/ETL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,4 +85,10 @@ They live in

```
lib/etl/runners/
```

The runners that that pull data from the NYC Open Data Portal and store an exact copy in the primary database live in

```
lib/etl/runners/primary_db_imports/
```
10 changes: 0 additions & 10 deletions docs/LOCAL_SETUP.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,16 +103,6 @@ DepartmentOfTransportation::BicycleCounter.import

See [nyc open data](YOUR_NYC_OPEN_DATA.md) for more information.

## Import Data into the Primary DB with Kiba

Import data into the primary database with Kiba.

```
DepartmentOfTransportation::BicycleCounter.run_import
```

See [nyc open data](YOUR_NYC_OPEN_DATA.md) for more information.

## Run ETL Workflows

Perform transformations on open data and store the results in the analytics database.
Expand Down
29 changes: 0 additions & 29 deletions docs/YOUR_NYC_OPEN_DATA.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,35 +35,6 @@ DepartmentOfTransportation::BicycleCounter.import(api_version: '3', content_type
DepartmentOfTransportation::BicycleCounter.import(api_version: '3', content_type: 'csv')
```

### run_import

The run_import method provides the same functionality as the import method.

It was implemented to demonstrate how to use the Kiba library to define/run ETL workflows.

Note: We are writing to the primary database here but in most cases Kiba will be used to write to the analytics database.

#### SODA2

```
DepartmentOfTransportation::BicycleCounter.run_import
DepartmentOfTransportation::BicycleCounter.run_import(api_version: '2', content_type: 'json')
```

```
DepartmentOfTransportation::BicycleCounter.run_import(api_version: '2', content_type: 'csv')
```

#### SODA3

```
DepartmentOfTransportation::BicycleCounter.run_import(api_version: '3', content_type: 'json')
```

```
DepartmentOfTransportation::BicycleCounter.run_import(api_version: '3', content_type: 'csv')
```

## Dataset Properties

The models contain the following class methods to describe a dataset's properties.
Expand Down
Loading