diff --git a/examples/active_fire_detection/README.md b/examples/active_fire_detection/README.md new file mode 100644 index 0000000..fb23a6e --- /dev/null +++ b/examples/active_fire_detection/README.md @@ -0,0 +1,172 @@ + +# Preparing Satellite Data for Wildfire Detection with Oxen.ai 🔥🛰️ + +## Introduction +Satellite imagery data is crucial to solving many of the worlds problems today. Wildfires is one such problem in which we can unleash the power of machine learning to help solve. Extreme wildfires can result in the loss of lives, ecosystems and entire communities. This costs societies billions of dollars to rebuild, and the earth years to recover from. + +With the help of machine learning and computer vision, we can help improve the detection of wildfires, enabling faster response times. Autonomous vehicles can also assist in suppressing high risk fires in dangerous places for humans to navigate. + +Luckily, there are few domains with greater quantity of high-quality, publicly accessible, data than satellite imagery. Government-funded missions such as [LANDSAT](https://www.usgs.gov/landsat-missions#:~:text=Since%201972%2C%20Landsat%20satellites%20have,natural%20resources%20and%20the%20environment.) (USGS / NASA) and [SENTINEL](https://sentinel.esa.int/web/sentinel/missions/sentinel-2) (ESA) have captured and transmitted terabytes of this data every day for multiple decades, providing an extremely rich resource to those wishing to develop models for pressing geospatial challenges like disaster response. + +![An image of an Ox fighting wildfires](images/wildfire-ox.png) + +While these datasets are information-rich and freely available, they are notoriously difficult to collaborate on and work with. + +## Making The Data More Accessible +In this tuturial, we will be working with a dataset that has meticulously annotated by researchers for space-based active wildfire detection. We'll be walking through how to prepare the [Landsat-8 Imagery](https://drive.google.com/drive/folders/1GIcAev09Ye4hXsSu0Cjo5a6BfL3DpSBm) (introduced in [this paper by Pereira et. al](https://arxiv.org/abs/2101.03409)) + +The data was originally stored in Google Drive, making it hard to explore and collaborate on, since it is a collection of zip files that need to be downloaded and processed. + +![A google drive of zip files](images/google_drive_zip_files.png) + +In this tutorial, we'll use tooling from [Oxen.ai](www.oxen.ai) to address the common pitfalls in working with satellite imagery. In doing so, we'll transform this widely-cited dataset for wildfires detection from a 10GB Google Drive-hosted .zip file collection to a cloneable, well-organized [Oxen.ai](www.oxen.ai) data repository ready for collaborative model development. + +TODO: Add a beautiful image of the oxen UI + +## The LANDSAT-8 Satellite Imagery Dataset + +This dataset contains patches of imagery extracted from LANDSAT-8 satellite imagery across Earth, many of which contain actively burning wildfires. + +While the authors processed the full 1.6TB of imagery captured by LANDSAT-8 in August 2020 (targeting peak fire season), we're primarily interested in the excellent subset of data for which they manually created segmentation masks denoting active burn. + +![An image showing extensive global landmass coverage of the LANDSAT-8 images included in the training set](images/landsat_patches.png) + +In addition to being a more manageable (12GB) dataset size for model fine-tuning and evaluation applications, these human-verified labels will give any downstream models the best possible signal with which to learn to detect fires in novel imagery. + + ![A synthetic color rendering of a landsat image of a fire zone, showing extensive smoke but partially obscured by cloud cover, side by side with the same image with a red pixel mask denoting the burn area](images/lansat_before_after.png) + + +To supplement these ground-truth manual annotations, the researchers include 5 auto-generated segmentation masks that don't require any direct human input. These are generated computationally, using algorithms previously developed and validated in the remote sensing literature. While less precise than the manual annotations, these can also be a useful reference point for model validation. + +The authors originally use a [GitHub repository](https://github.com/pereira-gha/activefire/tree/main) to distribute the dataset, but the only data currently hosted there are a 77-row metadata file called `images2020009.csv` (we'll come back to this shortly) and some download scripts to fetch the full dataset from .zip files Google Drive. + +![An image of the authors' GitHub repository for the project, showing a README with an outlink to a Google Drive repo](images/github_dataset.png) + + +### 🐂 Oxen is better than zip files +We see two primary areas in which we can better ready this dataset for collaborative model development. + +**1. Enable atomic changes.** If we want to add, modify, or remove 50 MB of data from this dataset, we should only have to push (and our colleagues subsequently pull) 50 MB — not the whole 10 GB zip file. + +If we identify a mistake and want to roll these changes back, we shouldn't have to hunt for the `DATA_VERSION_3_FINAL.zip` taking up extra storage on our drive - instead, we'll leverage Oxen's version control to seamlessly undo the change. + +**2. Improve data-label linking.** The source data and 6 human + computed label sets are currently linked somewhat tenuously through pattern matching in the image file paths. We'll nail this down to an authoritative, well-validated data -> labels mapping file to save our teammates some meticulous ETL work and speed up the model development process. + + +# TODO: Make the following more step-by-step-y + +Below feels pretty bloated and repetitive with what we talked about earlier. Can we just walk them through step by step how to transfer, with lightly touching on the benefits of each step? I imagine one or two sentences per step, and one image per step to keep the pace up and scannability high. + +### Making the data modeling-ready with Oxen + +#### 1. **Enabling atomic, versioned dataset collaboration** +We want to enable our team to efficiently iterate on this data throughout the duration of the wildfire detection project. In this dataset's original distribution via a collection of .zip files hosted on Google Drive, any changes, however small, to the source data would require a recompression and re-upload of the 10GB data folder. + +![An image showing an attempted download of an 8.9GB zip dump from Google Drive containing the data subset](images/google_drive_big_dataset.png) + +Integration of these changes by other team members would then require a re-download of the full dataset and manual merging with their local work, without any built-in awareness of the changes made between versions. + +Instead, we can unpack this data into an Oxen repository so that all our teammates can easily: +- Examine, query and explore the data in the OxenHub UI +- Clone it locally with `oxen clone https://www.hub.oxen.ai/ba/ActiveFire` +- Create a space-saving local working copy with `oxen checkout -b my-local-branch` +- Contribute to the improvement of the repo through adding, deleting, and modifying additional files without needing to re-package and re-push all 10GB of data. +- TODO: remote staging? or too much? + +**Early exploration and understanding** + +While waiting for the .zip files to download and unpack from the Google Drive source, we created a README referencing back to the original project. + +Included with the data was a tabular file titled `images202009.csv`—we couldn't quite tell what this file represented at first, so we pushed it up to Oxen as well to explore further. + +![An image of a tabular data file and its associated schema rendered in the OxenHub UI](images/oxenhub-tabular.png) +Already, we're getting a lot more information about the structures and datatypes of the dataset, while adding a version-controlled history we can turn to if we make any mistakes along the way. + +Diving a bit further into the images202009 file... +![An image of the OxenHub query UI with no natural language or SQL query yet filled out](images/oxenhub_blank_query.png) +...OxenHub gives us no-fuss insight into the shape and schema of the dataset, and the ability to explore it further to understand what it represents. + +The `productId` column ("LC08") and `cloudCover` tipped us off that this is a metadata export of full 5600x6100 pixel LANDSAT scenes from which the researchers exported their 256x256 pixel patches for modeling. + +![The OxenAI natural language query interface with "What is the distribution of processingLevel in this dataset? submitted in the box, and the resulting answer of 72 L1TP and 4 L1GT processing levels.](images/oxenhub_nlp_query.png) + +![The same query as above, translated into raw SQL via AI.](images/oxenhub_sql_query.png) + +We can use the natural language query interface to translate our analytic questions into SQL to reach a better understanding of which terrain correction methods are present in the dataset. The imbalance here between Precision and Terrain Correction (L1TP) and Systematic Terrain Correction (L1GT) is important to flag for our team during model and data development, as they could cause inconsistencies in the resulting data or confusion in the model training process. + +**Unpacking and restructuring** +Now that the Google Drive downloads have unzipped, we can `oxen add`, `oxen commit`, and then `oxen push` them to our repository, first in their original format... +![](images/oxenhub_pre_reorg.png) +...then, after a bit of friction, in a reorganized format that will set us up for clearer delineation and linkage between our data and labels: + +![](images/oxenhub_post_reorg.png) + + +#### 2. Improving data-label linking and training ergonomics +In the source dataset, the original satellite image for a patch, the authors' manually annotated labels, and the 5 sets of algorithmically determined labels are linked together only implicitly through commonalities in the filenames: + +``` +Input data (LANDSAT) path: +- LC08_L1GT_226074_20200921_20200921_01_RT_p00811 + +Manual annotation path: +- LC08_L1GT_226074_20200921_20200921_01_RT_v1_p00811 + +Kumar-Roy: +- LC08_L1GT_226074_20200921_20200921_01_RT_Kumar-Roy_p00811 + +Murphy: +- LC08_L1GT_226074_20200921_20200921_01_RT_Murphy_p00811 + +Intersection: +- LC08_L1GT_226074_20200921_20200921_01_RT_Intersection_p00811 + + +``` + +While this isn't particularly difficult to parse with regular expressions at training or inference time, it's also an opportune hiding spot for sneaky human error and unnecessary data munging overhead on each of the individual engineers working on this project. + +Further, while there are 9,044 total patches of raw imagery included in this dataset, the various annotation types vary in number and don't cover the same data: +- Manual annotations: 100 +- Kumar method: 391 +- Murphy method: 164 +- Schroeder method: 227 +- Intersection method: 118 +- Voting method: 198 + +...so solidifying awareness of coverage across the various labeling types in one place is key to effective collaboration on this dataset. + +**Creating a data -> label mapping file and validating with OxenHub** + +We parsed the filenames to the common patch_id, creating the following mapping file structure... + + +![A new data-to-labels mapper file where one row = one original LANDSAT scene, with columns for each of the 6 label masks and their paths](images/oxenhub_good_organization.png) +...associating each LANDSAT patch to the various label paths for that patch, when they exist. + +We can use schema metadata to let OxenHub know which columns are relative filepaths: + +``` +TODO: Schema setting from CLI +``` + +which quickly elucidates an error in our processing. + +**TODO IMAGE OF FAILED FILE EXISTENCE CHECK IN HUB** + +After hunting down the bug in our filepath processing code, we make a new commit with the fix and are ready to train! + +**TODO IMAGE OF SUCCESSFUL FILE EXISTENCE CHECKS IN HUB** + +This single source of truth for relating files across our repository will enable our team to more confidently investigate, sample, and evaluate on this dataset without having to worry about the integrity of the data-label linkage. + +### And we're off! +With these reorganizations, validations, and improvements, this dataset is now much more readily usable for collaborative model development for wildfire detection and segmentation applications. + +We'd love to see what kind of models you're building on top of it! Clone the repo [here](https://www.oxen.ai/ba/ActiveFire), reach out at [hello@oxen.ai](mailto:hello@oxen.ai), follow us on Twitter [@oxendrove](https://twitter.com/oxendrove?ref=blog.oxen.ai), dive deeper into the [documentation](https://github.com/Oxen-AI/oxen-release?ref=blog.oxen.ai), or **Sign up for Oxen today. [http://oxen.ai/register.](http://oxen.ai/register.?ref=blog.oxen.ai)** + +And remember—for every star on [GitHub](https://github.com/Oxen-AI/oxen-release?ref=blog.oxen.ai), an ox gets its wings. + +No, really...we hooked up an [Oxen repo](https://oxen.ai/ox/FlyingOxen?ref=blog.oxen.ai) to a GitHub web-hook that runs Stable Diffusion every time we get a star. [Go find yours!](https://oxen.ai/ox/FlyingOxen?ref=blog.oxen.ai) + +![An AI-generated winged ox image](images/ox.png) diff --git a/examples/active_fire_detection/code/format_csv.ipynb b/examples/active_fire_detection/code/format_csv.ipynb new file mode 100644 index 0000000..5fd266d --- /dev/null +++ b/examples/active_fire_detection/code/format_csv.ipynb @@ -0,0 +1,281 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "00acdb34", + "metadata": {}, + "source": [ + "## Label file creation + reorganization for ActiveFire dataset\n", + "\n", + "The output of this dataset is already included in the latest version of the repository, but to recreate, first clone the Oxen repository: \n", + "\n", + "```bash oxen clone https://hub.oxen.ai/ba/ActiveFire```\n", + "\n", + "Then run this file from the root of the ActiveFire directory." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7f590142", + "metadata": {}, + "outputs": [], + "source": [ + "import pandas as pd\n", + "import glob" + ] + }, + { + "cell_type": "code", + "execution_count": 93, + "id": "a7a4fa97", + "metadata": {}, + "outputs": [], + "source": [ + "# Glob existing files to get counts per directory after unpacking\n", + "landsat = glob.glob('data/landsat_patches/*')\n", + "manual = glob.glob('data/manual_annotations_patches/*')\n", + "masks = glob.glob('data/masks_patches/*')" + ] + }, + { + "cell_type": "code", + "execution_count": 94, + "id": "386513af", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Landsat patches: 9045\n", + "Manually annotated patches: 100\n", + "Masks annotated patches: 1098\n" + ] + } + ], + "source": [ + "print(f\"Landsat patches: {len(landsat)}\")\n", + "print(f\"Manually annotated patches: {len(manual)}\")\n", + "print(f\"Masks annotated patches: {len(masks)}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 95, + "id": "d2646109", + "metadata": {}, + "outputs": [], + "source": [ + "# Collect paths for each file / label type for matching to associate data with labels\n", + "data = pd.DataFrame()\n", + "landsat_paths = []\n", + "mask_paths = []\n", + "manual_paths = []\n", + "for path in landsat: \n", + " landsat_paths.append(path.split('/')[2])\n", + " \n", + "for path in manual:\n", + " manual_paths.append(path.split('/')[2])\n", + "\n", + "for path in masks:\n", + " mask_paths.append(path.split('/')[2])\n", + "\n", + "landsat_paths = pd.Series(landsat_paths)\n", + "manual_paths = pd.Series(manual_paths)\n", + "mask_paths = pd.Series(mask_paths)" + ] + }, + { + "cell_type": "code", + "execution_count": 96, + "id": "c7ab8fc1", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Kumar 391\n", + "Intersection 118\n", + "Murphy 164\n", + "Schroeder 227\n", + "Voting 198\n" + ] + } + ], + "source": [ + "# Separate out the 5 different non-manual masking methods and get their counts\n", + "kumar = mask_paths[mask_paths.str.contains('Kumar')]\n", + "intersection = mask_paths[mask_paths.str.contains('Intersection')]\n", + "murphy = mask_paths[mask_paths.str.contains('Murphy')]\n", + "schroeder = mask_paths[mask_paths.str.contains('Schroeder')]\n", + "voting = mask_paths[mask_paths.str.contains('Voting')]\n", + "\n", + "print(\"Kumar\", len(kumar))\n", + "print(\"Intersection\", len(intersection))\n", + "print(\"Murphy\", len(murphy))\n", + "print(\"Schroeder\", len(schroeder))\n", + "print(\"Voting\", len(voting))" + ] + }, + { + "cell_type": "code", + "execution_count": 97, + "id": "9c0061c2", + "metadata": {}, + "outputs": [], + "source": [ + "# Convert all to dataframes to make new columns\n", + "landsat = pd.DataFrame(landsat_paths)\n", + "kumar = pd.DataFrame(kumar)\n", + "intersection = pd.DataFrame(intersection)\n", + "murphy = pd.DataFrame(murphy)\n", + "schroeder = pd.DataFrame(schroeder)\n", + "voting = pd.DataFrame(voting)\n", + "manual = pd.DataFrame(manual_paths)\n" + ] + }, + { + "cell_type": "code", + "execution_count": 98, + "id": "103b83e5", + "metadata": {}, + "outputs": [], + "source": [ + "# Helper function to slice out the unique additive to each path, leaving only the matchable scene_ids\n", + "def get_scene_name(path, key):\n", + " path = path.replace(key, '')\n", + " return path \n", + "\n", + "# Landsat paths are just the scene ids, so no need to process\n", + "keys = {\n", + " 'kumar': 'Kumar-Roy_',\n", + " 'intersection': 'Intersection_',\n", + " 'murphy': 'Murphy_',\n", + " 'schroeder': 'Schroeder_',\n", + " 'voting': 'Voting_',\n", + " 'manual': 'v1_',\n", + " 'landsat': ''\n", + "}\n", + "\n", + "dfs = {'landsat': landsat, 'kumar': kumar, 'intersection': intersection, \n", + " 'murphy': murphy, \n", + " 'schroeder': schroeder, \n", + " 'voting': voting, \n", + " 'manual': manual}\n", + "\n", + "for key, df in dfs.items():\n", + " df['patch_id'] = df[0].apply(lambda x: get_scene_name(x, keys[key]))\n" + ] + }, + { + "cell_type": "code", + "execution_count": 99, + "id": "0c493942", + "metadata": {}, + "outputs": [], + "source": [ + "for key, df in dfs.items(): \n", + " df.rename(columns={0: f'{key}_path'}, inplace=True)" + ] + }, + { + "cell_type": "code", + "execution_count": 100, + "id": "3bee80cb", + "metadata": {}, + "outputs": [], + "source": [ + "# Merge all dataframes together on the common patch_id column, starting with landsat as the source of truth for left join\n", + "merged = dfs['landsat']\n", + "for key, df in dfs.items(): \n", + " if key != 'landsat':\n", + " merged = pd.merge(merged, df, on='patch_id', how='left')" + ] + }, + { + "cell_type": "code", + "execution_count": 101, + "id": "af16b5c7", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "✅ landsat count: 9045 equals original count of 9045\n", + "✅ kumar count: 391 equals original count of 391\n", + "✅ intersection count: 118 equals original count of 118\n", + "✅ murphy count: 164 equals original count of 164\n", + "✅ schroeder count: 227 equals original count of 227\n", + "✅ voting count: 198 equals original count of 198\n", + "✅ manual count: 100 equals original count of 100\n" + ] + } + ], + "source": [ + "# Ensure merge happened properly and all paths in sub-files matched a landsat file \n", + "for key, df in dfs.items():\n", + " merged_count = (merged[f'{key}_path'].isna() == False).sum()\n", + " original_count = len(df)\n", + " assert merged_count == original_count, f\"❌ Error with {key} merge. Merged count: {merged_count}, Original count: {original_count}\"\n", + " print(f\"✅ {key} count: {(merged[f'{key}_path'].isna() == False).sum()} equals original count of {len(df)}\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 102, + "id": "22f86baa", + "metadata": {}, + "outputs": [], + "source": [ + "# Create a scene id for merging with metadata before changing column identifiers \n", + "merged['scene_id'] = merged.landsat_path.apply(lambda x: \"_\".join(x.split('_')[:-1]))\n", + "\n", + "# Generate the full correct relative paths - these columns have different base folders\n", + "merged['landsat_path'] = merged.landsat_path.apply(lambda x: 'data/landsat_patches/'+x if x==x else x)\n", + "merged['manual_path'] = merged.manual_path.apply(lambda x: 'data/manual_annotations_patches/'+x if x==x else x)\n", + "\n", + "for key, df in dfs.items():\n", + " if key not in ['landsat', 'manual']:\n", + " merged[f'{key}_path'] = merged[f'{key}_path'].apply(lambda x: 'data/masks_patches/'+x if x==x else x)\n" + ] + }, + { + "cell_type": "code", + "execution_count": 103, + "id": "7ee76a36", + "metadata": {}, + "outputs": [], + "source": [ + "# Merge with scene-level metadata \n", + "metadata = pd.read_csv('labels/images202009.csv', sep=';')\n", + "final_merged = pd.merge(merged, metadata, left_on = 'scene_id', right_on='productId', how='left')\n", + "\n", + "final_merged.to_csv(\"labels/labels.csv\", drop=True)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/active_fire_detection/images/github_dataset.png b/examples/active_fire_detection/images/github_dataset.png new file mode 100644 index 0000000..b40ed5e Binary files /dev/null and b/examples/active_fire_detection/images/github_dataset.png differ diff --git a/examples/active_fire_detection/images/google_drive_big_dataset.png b/examples/active_fire_detection/images/google_drive_big_dataset.png new file mode 100644 index 0000000..58503c5 Binary files /dev/null and b/examples/active_fire_detection/images/google_drive_big_dataset.png differ diff --git a/examples/active_fire_detection/images/google_drive_data.png b/examples/active_fire_detection/images/google_drive_data.png new file mode 100644 index 0000000..0e689f1 Binary files /dev/null and b/examples/active_fire_detection/images/google_drive_data.png differ diff --git a/examples/active_fire_detection/images/google_drive_zip_files.png b/examples/active_fire_detection/images/google_drive_zip_files.png new file mode 100644 index 0000000..ddc2fe3 Binary files /dev/null and b/examples/active_fire_detection/images/google_drive_zip_files.png differ diff --git a/examples/active_fire_detection/images/landsat_burn_annotated.png b/examples/active_fire_detection/images/landsat_burn_annotated.png new file mode 100644 index 0000000..913b2ad Binary files /dev/null and b/examples/active_fire_detection/images/landsat_burn_annotated.png differ diff --git a/examples/active_fire_detection/images/landsat_burn_plain.png b/examples/active_fire_detection/images/landsat_burn_plain.png new file mode 100644 index 0000000..081fce7 Binary files /dev/null and b/examples/active_fire_detection/images/landsat_burn_plain.png differ diff --git a/examples/active_fire_detection/images/landsat_patches.png b/examples/active_fire_detection/images/landsat_patches.png new file mode 100644 index 0000000..c24b314 Binary files /dev/null and b/examples/active_fire_detection/images/landsat_patches.png differ diff --git a/examples/active_fire_detection/images/lansat_before_after.png b/examples/active_fire_detection/images/lansat_before_after.png new file mode 100644 index 0000000..42ef7ee Binary files /dev/null and b/examples/active_fire_detection/images/lansat_before_after.png differ diff --git a/examples/active_fire_detection/images/ox.png b/examples/active_fire_detection/images/ox.png new file mode 100644 index 0000000..7cc9ccb Binary files /dev/null and b/examples/active_fire_detection/images/ox.png differ diff --git a/examples/active_fire_detection/images/oxenhub-first.png b/examples/active_fire_detection/images/oxenhub-first.png new file mode 100644 index 0000000..2bbf7e3 Binary files /dev/null and b/examples/active_fire_detection/images/oxenhub-first.png differ diff --git a/examples/active_fire_detection/images/oxenhub-tabular.png b/examples/active_fire_detection/images/oxenhub-tabular.png new file mode 100644 index 0000000..d7ed065 Binary files /dev/null and b/examples/active_fire_detection/images/oxenhub-tabular.png differ diff --git a/examples/active_fire_detection/images/oxenhub_blank_query.png b/examples/active_fire_detection/images/oxenhub_blank_query.png new file mode 100644 index 0000000..521e237 Binary files /dev/null and b/examples/active_fire_detection/images/oxenhub_blank_query.png differ diff --git a/examples/active_fire_detection/images/oxenhub_first.png b/examples/active_fire_detection/images/oxenhub_first.png new file mode 100644 index 0000000..1c67cf2 Binary files /dev/null and b/examples/active_fire_detection/images/oxenhub_first.png differ diff --git a/examples/active_fire_detection/images/oxenhub_good_organization.png b/examples/active_fire_detection/images/oxenhub_good_organization.png new file mode 100644 index 0000000..e113e9d Binary files /dev/null and b/examples/active_fire_detection/images/oxenhub_good_organization.png differ diff --git a/examples/active_fire_detection/images/oxenhub_nlp_query.png b/examples/active_fire_detection/images/oxenhub_nlp_query.png new file mode 100644 index 0000000..b6440c5 Binary files /dev/null and b/examples/active_fire_detection/images/oxenhub_nlp_query.png differ diff --git a/examples/active_fire_detection/images/oxenhub_post_reorg.png b/examples/active_fire_detection/images/oxenhub_post_reorg.png new file mode 100644 index 0000000..fbe6659 Binary files /dev/null and b/examples/active_fire_detection/images/oxenhub_post_reorg.png differ diff --git a/examples/active_fire_detection/images/oxenhub_pre_reorg.png b/examples/active_fire_detection/images/oxenhub_pre_reorg.png new file mode 100644 index 0000000..a3c4649 Binary files /dev/null and b/examples/active_fire_detection/images/oxenhub_pre_reorg.png differ diff --git a/examples/active_fire_detection/images/oxenhub_sql_query.png b/examples/active_fire_detection/images/oxenhub_sql_query.png new file mode 100644 index 0000000..8f6ad9e Binary files /dev/null and b/examples/active_fire_detection/images/oxenhub_sql_query.png differ diff --git a/examples/active_fire_detection/images/patches_bad_wide.png b/examples/active_fire_detection/images/patches_bad_wide.png new file mode 100644 index 0000000..a68d9c6 Binary files /dev/null and b/examples/active_fire_detection/images/patches_bad_wide.png differ diff --git a/examples/active_fire_detection/images/patches_good_wide.png b/examples/active_fire_detection/images/patches_good_wide.png new file mode 100644 index 0000000..ebf02d9 Binary files /dev/null and b/examples/active_fire_detection/images/patches_good_wide.png differ diff --git a/examples/active_fire_detection/images/patches_good_zoom.png b/examples/active_fire_detection/images/patches_good_zoom.png new file mode 100644 index 0000000..deb55c3 Binary files /dev/null and b/examples/active_fire_detection/images/patches_good_zoom.png differ diff --git a/examples/active_fire_detection/images/wildfire-ox.png b/examples/active_fire_detection/images/wildfire-ox.png new file mode 100644 index 0000000..2a2f062 Binary files /dev/null and b/examples/active_fire_detection/images/wildfire-ox.png differ diff --git a/examples/automated_human_feedback_collection/code/app.py b/examples/automated_human_feedback_collection/code/app.py index 308a7a0..7f2df30 100644 --- a/examples/automated_human_feedback_collection/code/app.py +++ b/examples/automated_human_feedback_collection/code/app.py @@ -8,19 +8,19 @@ import shortuuid from oxen import RemoteRepo -# Config Oxen -repo = RemoteRepo("ba/active-learning-test") -repo.checkout("dev") # Config model training pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4") # pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-0.9", torch_dtype=torch.float16, use_safetensors=True, variant="fp16") pipe.to("cuda") +# Config Oxen +repo = RemoteRepo("ba/active-learning-test") +repo.checkout("dev") + IMAGE_DEST="images" NUM_IMAGES = 4 - def generate_images(prompt): images = pipe(prompt, guidance_scale=7.5, num_images_per_prompt=NUM_IMAGES).images return images