Comment on code changes#3
Conversation
5abe289 to
403b7ff
Compare
| @@ -1,116 +0,0 @@ | |||
| aiohttp==3.8.4 | |||
There was a problem hiding this comment.
Lets add import dependencies here. I can't install an environment that immediately works
There was a problem hiding this comment.
In addition, I think in order to work with spark, you'd need JDK installed. Were there any issues here for you?
| ``` | ||
| jupyter notebook | ||
| ``` | ||
|
|
There was a problem hiding this comment.
Lets add information about where to get the file data. And an explanation on how to query the data
| @@ -1,163 +0,0 @@ | |||
| { | |||
There was a problem hiding this comment.
It is not clear what imports notebook does.
There was a problem hiding this comment.
Presumably a place to import data. Let's make the beginning of the pipeline not depend on dune and load files.
There was a problem hiding this comment.
Lets solidify pipeline. Lets keep pipeline a sacred folder where we PR our changes.
Lets do a folder structure like this
- pipeline
- etl
- ingest (ideally this would be dune client- we are having issues here. this output to data/source. Ideally this would run indepdendently from others and slowly update data/source folder).
- transform (removing bots here. this output to data/clean)
- analysis (all of these run in parallel)
- model (these read from data/clean and output to data/analysis)
- ranking (these read from data/clean and output to data/analysis)
- etl
- web (this will be the graphical dashboard that reads from data for purpose for purpose of presentation. I wud say this operates similar to model and ranking but with the priority of showing particular things to the user for ur presentation. The streamlit app then can be https://docs.streamlit.io/streamlit-community-cloud/get-started/deploy-an-app deployed)
- data
- source
- clean
- analysis
There was a problem hiding this comment.
There are 2 reamining questions
- how we orchestrate this pipeline
- how we host the data/
- lets just put the files on github for now. Its easy access. I will make this repo public so u dont have any permission issues. try this and if we hit limitations then we can alternate to something else. Please let me know how this goes. The source folder of the parquet files is about 900MB after decompressing. Based on this I think we are ok for now.
| @@ -1,316 +0,0 @@ | |||
| { | |||
There was a problem hiding this comment.
There are 2 transformation files. Unfortunately, I didn't find any good prediction model for bots within an exchange ecosystem.
a194092 to
c7e00cf
Compare
1 participant
This PR is for me to comment on code changes. DO NOT MERGE