Data Job Market Intelligence

▶ Live dashboard · 📄 Project showcase

What skills are actually in demand for data roles, what do they pay, and how is that changing over time?

This project answers that question with data instead of guesswork. It runs an automated pipeline that collects data-job postings from public APIs, structures the messy real-world fields (skills, salaries) hidden in their text, stores them in a warehouse, and surfaces the trends. It is designed to keep running on a schedule, so the dataset get richer over time.

Why this project exists. Most data portfolios reuse the same clean Kaggle CSVs, which proves little about real work. Here the data is sourced from the wild, arrives messy, and has to be cleaned, parsed, and modeled end-to-end.

---

Key findings

From ~440 postings across 5 markets (Remote-international, US, Australia, Malaysia, Indonesia). Directional, not definitive which regenerate anytime with python -m src.report, and they firm up as the scheduled collector accumulates data.

• 🌏 Local vs. remote demand differs sharply. In Indonesia, roles lean analyst/BI SQL (77%), Python (73%), then Tableau/Power BI (~36%) and ETL/Airflow (data-engineering). Remote-international roles lean Python-first (57%) with more ML and cloud (AWS, Git). Different skill prep depending on which market you target.
• 📊 Across all markets, SQL + Python dominate; everything else trails. A BI tool (Tableau/Power BI) is close to mandatory for local analyst roles.
• 💸 Specialized ML skills carry a pay premium (remote USD salaries): Deep Learning / PyTorch / TensorFlow roughly $150–170k vs roughly $90–116k for general analytics tooling. (Small per-skill samples; salary data is scarce outside remote-USD while local postings rarely advertise pay.)

Caveat: English-language postings; salary stats cover only the remote-USD subset that advertises pay. See Data & limitations.

Charts

Generated by python -m src.figures; the full narrative walk-through lives in notebooks/analysis.ipynb.

Demand	Pay by skill
Roles compared	Skill co-occurrence

Salary model (Phase 4)

Predicting advertised salary, done the honest way, a small regularised model, 5-fold cross-validation, and transparent limitations. On the current ~60 advertised-salary postings:

Model	MAE (CV)	Notes
Baseline (global median)	~$42k	the bar every model must clear
RidgeCV (skills + role + seniority)	~$39k	interpretable coefficients
RandomForest	~$37k	best, ~13% better than baseline

R² hovers near zero which is an honest signal that advertised salary is hard to predict from a handful of features on this little data. The model's value is the pipeline: it sharpens automatically as the collector accumulates more salaries.

Interpretation (RidgeCV). Seniority (+$22k), Machine Learning (+$19k) and Deep Learning (+$11k) push predicted pay up. SQL and cloud show negative coefficients not because they lower pay, but because in this small sample they cluster in lower-paying analyst roles (a confounding artifact the writeup calls out explicitly). Directional, not causal.

Interactive dashboard (Phase 5)

Windows: just double-click run.bat. Otherwise:

python -m streamlit run app.py

If streamlit run app.py errors with No module named 'streamlit.cli', a stale launcher from another Python install is shadowing it, use python -m streamlit run app.py (and run.bat).

A Streamlit app (app.py) that reuses the same analysis/model code, so it never drifts from the report. It has filters (role, source, work type), the four charts, a sortable posting explorer, and a live salary estimator.

Deploy it free (gives you a public URL for your résumé):

1. Push this repo to GitHub (the small warehouse .parquet files are committed on purpose so the app has data).
2. Go to share.streamlit.io, connect the repo, and set the main file to app.py.
3. Streamlit Community Cloud installs requirements.txt and serves the app at a shareable link. (Adzuna keys aren't needed to view it; add them as Streamlit "secrets" only if you want it to collect fresh data.)

---

How it works

 Public job APIs            This pipeline                      Outputs
┌───────────────┐     ┌──────────────────────────┐     ┌───────────────────┐
│  Remotive     │──▶  │ collect → normalize →     │ ──▶ │ data/raw/*.jsonl   │ (audit trail)
│  Jobicy       │     │ parse salaries → dedupe   │     │ warehouse/         │
│  Adzuna       │     │                           │     │   jobs.parquet     │ (analysis-ready)
└───────────────┘     └──────────────────────────┘     └───────────────────┘

• Sources (src/sources/) — one small module per job board, all mapped onto a single normalized schema. Adding a board is ~30 lines.
• Salary parsing (src/salary.py) — turns free text like "$80k - $100k" or "$18/hr" into (min, max, currency, period). The gnarly part, so it has its own unit tests.
• Skill extraction (src/skills/) — a curated taxonomy of ~70 data skills + a matcher that resolves ambiguous names (R vs "R&D", Java vs JavaScript, Excel vs the verb). Output: a tidy job_skills table, one row per posting × skill.
• Storage (src/storage.py) — immutable raw snapshots for reproducibility + a deduplicated parquet warehouse that tracks when each posting was first_seen / last_seen.
• Insights (src/report.py) — skill demand and median advertised pay per skill (advertised-USD-annual only, small samples gated).

Quickstart

# 1. Install (a virtual environment is recommended)
python -m venv .venv
.venv\Scripts\activate          # Windows
# source .venv/bin/activate     # macOS/Linux
pip install -r requirements.txt

# 2. (Optional) enable Adzuna for non-remote, higher-volume data:
#    copy .env.example -> .env and paste your free Adzuna keys.
#    Remotive + Jobicy need no keys, so you can skip this and still run.

# 3. Collect postings (writes data/warehouse/jobs.parquet)
python -m src.collect

# 4. Extract skills + print the demand & salary report
python -m src.extract_skills

# 5. Render the charts (PNGs into reports/figures/)
python -m src.figures

# 6. Train + cross-validate the salary model (baseline vs models)
python -m src.model

# 7. Launch the interactive dashboard
python -m streamlit run app.py    # (more robust than `streamlit run` if you
                                  #  have multiple Python installs)

# Anytime: peek at raw data, re-print insights
python -m src.peek
python -m src.report

For tests and the notebook, install the dev extras:

pip install -r requirements-dev.txt
pytest
jupyter notebook notebooks/analysis.ipynb

Re-run python -m src.collect whenever you like. It tops up the warehouse and deduplicates automatically. Re-run python -m src.extract_skills after each collection to refresh the skills table and insights. Tune what gets collected in config.yaml.

Project status / roadmap

• Phase 1 — Collection. Multi-source pipeline, normalized schema, salary parser + tests, deduplicated warehouse, raw audit snapshots.
• Phase 2 — Skill extraction. Curated taxonomy (~70 skills) + a matcher that handles ambiguous names (R vs "R&D", Java vs JavaScript, Excel vs the verb); tidy job_skills table; demand + salary-by-skill report.
• Phase 3 — Analysis & insights. Demand, pay-by-skill, role comparison and skill co-occurrence as a text report (src/report.py), reproducible charts (src/figures.py), and a narrative notebook.
• Phase 4 — Salary model. Baseline-first, RidgeCV + RandomForest, 5-fold cross-validation, interpretable coefficients, honest error reporting (src/model.py).
• Phase 5 — Dashboard. Interactive Streamlit app (app.py) with filters, charts, a posting explorer and a live salary estimator; one-click deployable to Streamlit Community Cloud.
• Phase 6 — Automation. Scheduled refresh via GitHub Actions.

Data & limitations

• Sources & markets — every posting is tagged with a market so you can compare e.g. Indonesia vs Remote-international vs Australia:

  Source	Market(s)	Strength	Caveat
  Remotive, Jobicy	Remote (international)	real advertised USD salaries	remote-only
  RemoteOK	Remote (international)	popular board, free	tags are SEO-spam (we title-filter)
  Adzuna	US, Australia (any of its countries)	volume, geography, on-site roles	salaries mostly estimates; no Indonesia/Malaysia
  Jooble	Indonesia, Malaysia	local coverage exists	very thin for data roles; no skills/salary
  JSearch	any country incl. Indonesia	aggregates Google for Jobs (LinkedIn/Indeed/Glassdoor)	needs RapidAPI key; free tier rate-limited

• The local-data reality — free APIs barely cover the Indonesian data-job market (Jooble returns ~1 real Indonesian data role). JSearch is the only realistic path to meaningful local data, since it taps the big boards that actually carry those postings.
• Predicted vs advertised salary — Adzuna fills in an estimated salary (salary_is_predicted = true) when a posting doesn't state one. Those are flagged and excluded from salary modeling, or you'd be modeling Adzuna's model. Only ~⅓ of salaried postings carry a genuinely advertised number.
• Coverage bias — still skews English-language and tech-forward; remote vs on-site is now roughly balanced thanks to Adzuna. Treat findings as directional, not census-accurate.
• Skill extraction is dictionary-based — high precision (few false hits) but bounded recall: a skill only counts if it's in the taxonomy. Add entries to widen coverage.
• Description truncation is surfaced, not hidden. Some sources return short teasers (Adzuna averages ~500 chars vs ~2,400–6,300 for Jobicy/JSearch/RemoteOK), so a missing skill there is missing data, not a true "not required". Rather than silently drop that data, the report and dashboard show a per-source quality audit (quality_by_source), and demand can optionally be recomputed over detailed postings only (the rich_only toggle). Adzuna still earns its place for volume, geography and on-site coverage.
• Ethics — official JSON APIs (no scraping behind logins), modest request volumes, public posting fields only. API keys live in a git-ignored .env.

Configuring sources

Toggle sources and search terms in config.yaml. Adzuna needs a free app_id / app_key from https://developer.adzuna.com/. Once you have them, copy .env.example to .env and paste them in. Adding a brand-new job board is ~30 lines: subclass BaseSource in src/sources/ and register it.

Tech

Python · requests · pandas / pyarrow (parquet) · BeautifulSoup · scikit-learn · matplotlib · streamlit · Jupyter · pytest. Planned: GitHub Actions (Phase 6).