Seed RAG corpus with EmpatheticDialogues + Social Chemistry 101 datasets

[Copilot]

Two open datasets (EmpatheticDialogues — 25K emotional conversations; Social Chemistry 101 — 292K social norm rules-of-thumb) provide empathetic and norm-aware grounding for the Ask a Sailor RAG agent without fine-tuning.

Ingestion scripts

• scripts/ingest_empathetic_dialogues.py — Parses CSV splits, groups utterances by conv_id, labels Speaker/Listener, replaces _comma_ tokens, filters PII, emits JSONL chunks keyed by emotion.
• scripts/ingest_social_chemistry.py — Parses TSV, converts each rule-of-thumb into a chunk with situation/action/judgment, filters PII, emits JSONL.

Both output the same chunk schema used by the existing corpus (chunk_id, doc_id, source_type, text, metadata).

Agent --sources flag

AskASailorAgent accepts an optional sources list. The CLI exposes --sources as comma-separated external corpus names:

python agent.py --club lyc --sources empathetic_dialogues,social_chemistry --question "..."

External corpora are loaded from corpus_dir/{source}/corpus.jsonl alongside club corpora. Missing sources warn, don't crash.

Other

• data/external/ added to .gitignore
• 22 new tests: chunk format validation, PII filtering, chunk counts, agent source loading, graceful missing-source handling

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[ask-a-sailor] Seed RAG corpus with EmpatheticDialogues + Social Chemistry 101 datasets</issue_title>
<issue_description>Two open datasets dramatically improve empathetic, norm-aware responses without fine-tuning: (1) EmpatheticDialogues (facebook/empathetic_dialogues on HuggingFace) -- 25K conversations grounded in 32 emotional situations, CC license, 25MB download. wget https://dl.fbaipublicfiles.com/parlai/empatheticdialogues/empatheticdialogues.tar.gz. Gives the model emotional grounding for parent Q&A. (2) Social Chemistry 101 (github.com/mbforbes/social-chemistry-101) -- 292K rules of thumb about social and moral norms, CC BY-SA 4.0. storage.googleapis.com/ai2-mosaic-public/projects/social-chemistry/data/social-chem-101.zip. Provides the unwritten social rules ASD families need made explicit. Work: (1) Download both datasets to a new data/external/ directory (gitignored), (2) write an ingestion script scripts/ingest_empathetic_dialogues.py and scripts/ingest_social_chemistry.py that convert each to the corpus JSONL chunk format, (3) add these as optional corpus sources configurable via --sources flag on AskASailorAgent. Tests: verify chunk format, verify no PII, verify chunk count.</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes [ask-a-sailor] Seed RAG corpus with EmpatheticDialogues + Social Chemistry 101 datasets #17

---

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

[fullharbor]

@copilot curious i have several of these "download from cloud" chat sessions. Are they incorperated, or not?

[Copilot]

@copilot curious i have several of these "download from cloud" chat sessions. Are they incorperated, or not?

The download is not built into the scripts — it's a separate manual step. Each script's docstring documents the wget/unzip commands to run first, then you point the script at the local directory with --input-dir. The scripts only handle the parse → JSONL conversion + PII filtering.

If you'd like, I can add a --download flag to each script that auto-fetches the dataset when the input directory doesn't exist. Let me know.