Feature/ingestion - #9
Open
ChunEnLin wants to merge 18 commits into
Open
Conversation
…eorganize_admin_data
…eorganize_admin_data (ruff check)
YenCheng1226
requested changes
Feb 10, 2026
YenCheng1226
left a comment
YenCheng1226
left a comment
Collaborator
There was a problem hiding this comment.
資料處理的流程目前都沒有問題！我有幫你新增了資料處理核心function 的 unit test ，另外GEE 的環境依賴問題正好提醒我，之後我們需要改成將使用者下載 GEE 的 GADM 資料到本地寫成另外的function，ingestion.py 只處理 gadm input 和 output 的問題，這個問題我們下次開會會討論，有沒有更好的解法
Comment on lines
+12
to
+20
| # Initialize GEE (in order to access GAUL dataset in GEE) | ||
| def initialize_gee(gee_project): | ||
| try: | ||
| ee.Initialize(project=gee_project) | ||
| print("Earth Engine initialized successfully.") | ||
| except Exception: | ||
| print("you need to authenticate GEE:") | ||
| ee.Authenticate() # Need to authenticate only once | ||
| ee.Initialize(project=gee_project) |
Collaborator
There was a problem hiding this comment.
initialization 目前是硬編碼指定的 gee_project ，團隊協作如果成員的 gee_project 名稱不一樣，run 這份 code 就會出 bug ，也提醒我 initialization 應該另外寫成 module，這部分我之後會優先處理
| # Main function | ||
| def main(): | ||
| # Filepath settings & Google Earth Engine project name setting | ||
| input_filepath = "/home/chunen/nas/HAZAMA_data/public_emdat_custom_request_2026-01-28.csv" # noqa: E501 |
Collaborator
There was a problem hiding this comment.
這邊需要改成不依賴既有路徑和環境的方式，情境是：如果是新的使用者使用，他要能重新下載到本地端進行處理，只需要指定的python 環境就可以run
| # Filepath settings & Google Earth Engine project name setting | ||
| input_filepath = "/home/chunen/nas/HAZAMA_data/public_emdat_custom_request_2026-01-28.csv" # noqa: E501 | ||
| # output_filepath = "/home/chunen/HAZAMA/HAZAMA/outputs/data_ingestion.csv" | ||
| my_gee_project = "oceanic-hash-467505-r2" |
Collaborator
There was a problem hiding this comment.
前面有說到，這裡hard-coding成個人的 gee_project ，之後其他人進來接者開發會有依賴問題
| # output_filepath = "/home/chunen/HAZAMA/HAZAMA/outputs/data_ingestion.csv" | ||
| my_gee_project = "oceanic-hash-467505-r2" | ||
| # GADM GeoPackage filepath setting | ||
| gadm_filepath = "/home/chunen/nas/HAZAMA_data/gadm_410-levels-ADM2.gpkg" # noqa: E501 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
差不多完成了，但有幾個部分需要在使用者執行前去修改：
1.input_filepath
2.my_gee_project
3.gadm_filepath
另外，目前是設定只會跑前100筆資料
df_to_process = df_processed.iloc[0:100].copy()
另外，pyproject.toml 的依賴套件有 geemap，但這個我只有用在 demos/ingestion_test.ipynb 裡面，src/ingestion.py 並沒有用到這個套件