diff --git a/AGENTS.md b/AGENTS.md index 5122d1b4..67127edb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -55,7 +55,8 @@ This guide explains how automation agents and human contributors should work wit - **Routes** – `GetRoutes`, `GetRoutesMinimal`. The minimal variant returns `RouteMinimalResponse` with deduplicated `LineMinimal` data; paging tokens are currently empty (pagination not implemented). - **Train types** – `GetTrainTypesByStationId`, `GetRouteTypes`. Train types aggregate by line group and include related lines plus optional train type metadata. Rail variants use `TrainTypeKind::{Default, Branch, Rapid, Express, LimitedExpress, HighSpeedRapid, CommuterRapid}` (0-6); bus variants use `BusRoute` (7), which represents a `(route_id, shape_id)` operation pattern (e.g. 循環 / 短ターン / 支線) generated automatically from the configured GTFS bus feeds (Toei Bus, Seibu Bus, Keio Bus) and the converted Tokyu Bus JSON. - **GTFS bus integration** – At startup, `src/import.rs::integrate_gtfs_to_stations()` ingests GTFS feeds into `gtfs_*` tables and then projects them onto the shared `stations` / `lines` / `types` / `station_station_types` tables. Every configured GTFS feed is imported, including Seibu Bus and Keio Bus (both downloaded from ODPT with `ODPT_ACCESS_TOKEN`). Tokyu Bus ordinary-route `BusroutePattern`, `BusstopPole`, and `BusTimetable` JSON are converted into the same `gtfs_*` representation; pattern IDs become `shape_id` values so route variants remain queryable as bus TrainTypes. The Tokyu-operated Ota, Shinagawa, and Meguro community buses use their official GTFS feeds and matching JSON routes are excluded to prevent duplicates. `ODPT_ACCESS_TOKEN` is required for authenticated sources. Stops whose Tokyu JSON records omit coordinates remain available to name and route queries but not coordinate searches. `transport_type` (0: rail, 1: bus) on both `stations` and `lines` keeps rail and bus records queryable side by side. GTFS IDs are namespaced per feed before import to avoid cross-operator collisions. `line_cd` (100,000,000+), `station_cd` / `station_g_cd` (200,000,000+), and bus `type_cd` / `line_group_cd` (100,000,000+) are all deterministic fnv1a hashes that stay clear of the rail data ranges. Disable the entire bus pipeline with `DISABLE_BUS_FEATURE=true`. -- **Bus English-name fallback** – Keio Bus GTFS and Tokyu Bus JSON ship a kana reading for every stop but no English (`en`) name, so `gtfs_stops.stop_name_r` would otherwise be empty. `src/domain/romaji.rs::romaji_display_name()` derives a modified-Hepburn romanization (with macrons for long vowels, matching the curated rail style: Tōkyō / Kyōto / Shin-Ōsaka) from that reading, and `import.rs` fills `stop_name_r` with it whenever the feed provides no `en` value. Because `stop_name_r` is the single upstream source that fans out into the `stations` projection, `search_by_name`, and the romanized bus route/headsign names, this supplements every English-facing surface at once. The fallback never overwrites a real `en` value, and a reading with no convertible kana stays `NULL` rather than emitting a partial transcription. When projecting into `stations`, `station_name_rn` is filled with the plain-ASCII spelling via `romaji::strip_macrons()` (Tōkyō → Tokyo), mirroring the rail dataset's `_r` (macron) / `_rn` (macron-free) column pair. +- **Bus stop translations (readings & English)** – GTFS-JP `translations.txt` layouts differ per feed, so `load_gtfs_translations` resolves columns by header name (Seibu ships 6 columns without `record_sub_id`; Keio and the Tokyu community feeds ship 7) and indexes each `stop_name` translation under both keys it may use: `record_id` (== the stop_id, Seibu — with the "-NN" pole suffix also mapped to the parent stop_id) and `field_value` (== the Japanese stop_name, Keio / Tokyu community, where `record_id` is left empty). `import_gtfs_stops` then looks a stop's translation up by stop_id first, then by name. Keying only by `record_id` (the previous behavior) silently dropped every field_value-keyed feed, leaving `station_name_k` filled with the kanji stop_name and `station_name_r` empty. Readings arriving as half-width katakana (`ﾆｼﾊﾁｵｳｼﾞ`, Keio / Tokyu community) are folded to full-width via `romaji::to_fullwidth_katakana()` before storage. +- **Bus English-name fallback** – When a feed provides no English (`en`) translation for a stop — e.g. Tokyu Bus ordinary-route JSON, which carries only `dc:title` and `odpt:kana` — `src/domain/romaji.rs::romaji_display_name()` derives a modified-Hepburn romanization (with macrons for long vowels, matching the curated rail style: Tōkyō / Kyōto / Shin-Ōsaka) from the kana reading, and `import.rs` fills `stop_name_r` with it. The fallback never overwrites a real `en` value, and a reading with no convertible kana stays `NULL` rather than emitting a partial transcription. Because `stop_name_r` is the single upstream source that fans out into the `stations` projection, `search_by_name`, and the romanized bus route/headsign names, this supplements every English-facing surface at once. When projecting into `stations`, `station_name_rn` is filled with the plain-ASCII spelling via `romaji::strip_macrons()` (Tōkyō → Tokyo), mirroring the rail dataset's `_r` (macron) / `_rn` (macron-free) column pair. - **TTS metadata** – `Station`, `StationMinimal`, `Line`, and `TrainType` expose `name_ipa` / `name_roman_ipa` plus `name_tts_segments` for multi-segment pronunciation output. Use `name_tts_segments` when clients need per-token SSML construction for mixed-language names such as `Kasai-Rinkai Park`. - **Connected routes** – `GetConnectedRoutes`. `QueryInteractor::get_connected_stations` is not implemented yet and returns an empty vector; update the use-case and infrastructure layers together when adding real logic. - Changes to the service contract require coordinated updates to `proto/stationapi.proto`, regenerated code via `tonic-build`, and corresponding adjustments in both presentation and use-case layers. diff --git a/stationapi/src/domain/romaji.rs b/stationapi/src/domain/romaji.rs index 48878ee4..50f42d4b 100644 --- a/stationapi/src/domain/romaji.rs +++ b/stationapi/src/domain/romaji.rs @@ -257,18 +257,148 @@ fn is_separator(c: char) -> bool { matches!(c, '\u{30FB}' | ' ' | '\u{3000}' | '/' | '\u{FF0F}') } -/// Normalize hiragana to katakana so both reading styles convert identically. +/// Normalize a kana string to full-width katakana so every reading style +/// converts identically. Delegates to [`to_fullwidth_katakana`]. fn normalize_kana(input: &str) -> String { - input - .chars() - .map(|c| { - if ('\u{3041}'..='\u{3096}').contains(&c) { - char::from_u32(c as u32 + 0x60).unwrap_or(c) - } else { - c + to_fullwidth_katakana(input) +} + +/// Normalize hiragana and half-width katakana to full-width katakana. +/// +/// Some ODPT bus GTFS feeds ship their `ja-Hrkt` readings as half-width +/// katakana with combining voiced/semi-voiced marks (e.g. `ﾆｼﾊﾁｵｳｼﾞ`), which +/// must be folded to full-width before they can be stored in the katakana +/// column or romanized. Full-width katakana passes through unchanged, so the +/// function is idempotent. +pub fn to_fullwidth_katakana(input: &str) -> String { + let chars: Vec = input.chars().collect(); + let mut out = String::with_capacity(chars.len()); + let mut i = 0; + while i < chars.len() { + let c = chars[i]; + + // Hiragana → katakana (same +0x60 offset used across the module). + if ('\u{3041}'..='\u{3096}').contains(&c) { + out.push(char::from_u32(c as u32 + 0x60).unwrap_or(c)); + i += 1; + continue; + } + + // Half-width katakana → full-width, folding a trailing voiced (ﾞ) or + // semi-voiced (ﾟ) mark into the preceding kana. + if let Some(base) = halfwidth_katakana_to_fullwidth(c) { + match chars.get(i + 1).copied() { + Some('\u{FF9E}') => { + if let Some(voiced) = apply_dakuten(base) { + out.push(voiced); + i += 2; + continue; + } + } + Some('\u{FF9F}') => { + if let Some(semi) = apply_handakuten(base) { + out.push(semi); + i += 2; + continue; + } + } + _ => {} } - }) - .collect() + out.push(base); + i += 1; + continue; + } + + out.push(c); + i += 1; + } + out +} + +/// Map a half-width katakana code point to its full-width base form. +fn halfwidth_katakana_to_fullwidth(c: char) -> Option { + let full = match c { + '\u{FF65}' => '\u{30FB}', // ・ middle dot + 'ｦ' => 'ヲ', + 'ｧ' => 'ァ', + 'ｨ' => 'ィ', + 'ｩ' => 'ゥ', + 'ｪ' => 'ェ', + 'ｫ' => 'ォ', + 'ｬ' => 'ャ', + 'ｭ' => 'ュ', + 'ｮ' => 'ョ', + 'ｯ' => 'ッ', + 'ｰ' => 'ー', + 'ｱ' => 'ア', + 'ｲ' => 'イ', + 'ｳ' => 'ウ', + 'ｴ' => 'エ', + 'ｵ' => 'オ', + 'ｶ' => 'カ', + 'ｷ' => 'キ', + 'ｸ' => 'ク', + 'ｹ' => 'ケ', + 'ｺ' => 'コ', + 'ｻ' => 'サ', + 'ｼ' => 'シ', + 'ｽ' => 'ス', + 'ｾ' => 'セ', + 'ｿ' => 'ソ', + 'ﾀ' => 'タ', + 'ﾁ' => 'チ', + 'ﾂ' => 'ツ', + 'ﾃ' => 'テ', + 'ﾄ' => 'ト', + 'ﾅ' => 'ナ', + 'ﾆ' => 'ニ', + 'ﾇ' => 'ヌ', + 'ﾈ' => 'ネ', + 'ﾉ' => 'ノ', + 'ﾊ' => 'ハ', + 'ﾋ' => 'ヒ', + 'ﾌ' => 'フ', + 'ﾍ' => 'ヘ', + 'ﾎ' => 'ホ', + 'ﾏ' => 'マ', + 'ﾐ' => 'ミ', + 'ﾑ' => 'ム', + 'ﾒ' => 'メ', + 'ﾓ' => 'モ', + 'ﾔ' => 'ヤ', + 'ﾕ' => 'ユ', + 'ﾖ' => 'ヨ', + 'ﾗ' => 'ラ', + 'ﾘ' => 'リ', + 'ﾙ' => 'ル', + 'ﾚ' => 'レ', + 'ﾛ' => 'ロ', + 'ﾜ' => 'ワ', + 'ﾝ' => 'ン', + _ => return None, + }; + Some(full) +} + +/// Apply a dakuten (voiced mark) to a full-width katakana base, if voiceable. +fn apply_dakuten(c: char) -> Option { + let voiced = match c { + 'ウ' => 'ヴ', + 'カ' | 'キ' | 'ク' | 'ケ' | 'コ' | 'サ' | 'シ' | 'ス' | 'セ' | 'ソ' | 'タ' | 'チ' + | 'ツ' | 'テ' | 'ト' | 'ハ' | 'ヒ' | 'フ' | 'ヘ' | 'ホ' => { + char::from_u32(c as u32 + 1)? + } + _ => return None, + }; + Some(voiced) +} + +/// Apply a handakuten (semi-voiced mark) to a full-width katakana base. +fn apply_handakuten(c: char) -> Option { + match c { + 'ハ' | 'ヒ' | 'フ' | 'ヘ' | 'ホ' => char::from_u32(c as u32 + 2), + _ => None, + } } /// Two-character katakana combinations (palatalised sounds and loanword kana). @@ -522,6 +652,28 @@ mod tests { assert_eq!(romaji("とうきょう"), "tōkyō"); } + #[test] + fn halfwidth_katakana_is_folded_to_fullwidth() { + // Real Keio Bus ja-Hrkt readings (half-width with combining dakuten). + assert_eq!( + to_fullwidth_katakana("ﾆｼﾊﾁｵｳｼﾞｴｷﾐﾅﾐｸﾞﾁ"), + "ニシハチオウジエキミナミグチ" + ); + assert_eq!(to_fullwidth_katakana("ｱｷﾊﾞｼﾞﾝｼﾞｬ"), "アキバジンジャ"); + assert_eq!(to_fullwidth_katakana("ｱｻｶﾞﾔｼﾞｭｳﾀｸ"), "アサガヤジュウタク"); + // Handakuten and prolonged mark. + assert_eq!(to_fullwidth_katakana("ﾊﾟｰｸ"), "パーク"); + // Full-width katakana and hiragana are unchanged / folded (idempotent). + assert_eq!(to_fullwidth_katakana("トウキョウ"), "トウキョウ"); + assert_eq!(to_fullwidth_katakana("しぶや"), "シブヤ"); + } + + #[test] + fn halfwidth_reading_romanizes_end_to_end() { + // Keio Bus 阿佐ヶ谷住宅東 reading → Hepburn (used only when `en` is absent). + assert_eq!(romaji("ｱｻｶﾞﾔｼﾞｭｳﾀｸﾋｶﾞｼ"), "asagayajūtakuhigashi"); + } + #[test] fn separators_become_spaces() { assert_eq!(romaji("シブヤ・エキ"), "shibuya eki"); diff --git a/stationapi/src/import.rs b/stationapi/src/import.rs index 5770c740..f270bb0a 100644 --- a/stationapi/src/import.rs +++ b/stationapi/src/import.rs @@ -5,7 +5,7 @@ use serde::de::{DeserializeOwned, DeserializeSeed, SeqAccess, Visitor}; use serde::{Deserialize, Serialize}; use sqlx::{Connection, PgConnection}; use stationapi::config::fetch_database_url; -use stationapi::domain::romaji::{romaji_display_name, strip_macrons}; +use stationapi::domain::romaji::{romaji_display_name, strip_macrons, to_fullwidth_katakana}; use std::collections::{HashMap, HashSet}; use std::fmt; use std::fs::File; @@ -1151,72 +1151,84 @@ fn load_gtfs_translations( let mut rdr = ReaderBuilder::new().from_path(&translations_path)?; + // The GTFS-JP translations.txt column layout differs between feeds: Seibu + // omits `record_sub_id` (6 columns), while Keio and the Tokyu community feeds + // include it (7 columns). Resolve every column by header name so positions + // stay correct regardless of layout. + let headers = rdr.headers()?.clone(); + let column = |name: &str| headers.iter().position(|h| h == name); + let (Some(table_idx), Some(field_idx), Some(lang_idx), Some(text_idx)) = ( + column("table_name"), + column("field_name"), + column("language"), + column("translation"), + ) else { + warn!("translations.txt missing required columns; skipping translations."); + return Ok(translations); + }; + let record_id_idx = column("record_id"); + let field_value_idx = column("field_value"); + for result in rdr.records() { let record = result?; - // table_name,field_name,language,translation,record_id,record_sub_id,field_value - let table_name = record.get(0).unwrap_or(""); - let field_name = record.get(1).unwrap_or(""); - let language = record.get(2).unwrap_or(""); - let translation_text = record.get(3).unwrap_or(""); - let record_id = record.get(4).unwrap_or(""); + let cell = |idx: Option| idx.and_then(|i| record.get(i)).unwrap_or(""); // Only process stop_name translations for now - if table_name == "stops" && field_name == "stop_name" { - // Store translation for the exact record_id (e.g., "0001-01") - let key = ("stops".to_string(), record_id.to_string()); - let entry = translations.entry(key).or_default(); - - match language { - "ja" => entry.ja = Some(translation_text.to_string()), - "ja-Hrkt" => entry.ja_hrkt = Some(translation_text.to_string()), - "en" => entry.en = Some(translation_text.to_string()), - "zh-Hans" | "zh-Hant" | "zh" => entry.zh = Some(translation_text.to_string()), - "ko" => entry.ko = Some(translation_text.to_string()), - _ => {} - } + if record.get(table_idx) != Some("stops") || record.get(field_idx) != Some("stop_name") { + continue; + } - // Also store translation for parent stop_id (without suffix like "-01", "-02") - // This allows parent stops (location_type=1) to find translations + let language = record.get(lang_idx).unwrap_or(""); + let text = record.get(text_idx).unwrap_or(""); + let record_id = cell(record_id_idx); + let field_value = cell(field_value_idx); + + // Feeds key stop_name translations either by `record_id` (Seibu — the + // stop_id, sometimes with a "-01" pole suffix) or by `field_value` (Keio, + // Tokyu community — the Japanese stop_name itself, with record_id left + // empty). Index whichever the row provides so the stop importer can look + // the translation up by stop_id or by name. + if !record_id.is_empty() { + set_translation_language(&mut translations, record_id, language, text, true); + // Parent stop_id (drop the "-NN" pole suffix); first child wins. if let Some(parent_id) = record_id.rfind('-').map(|pos| &record_id[..pos]) { - let parent_key = ("stops".to_string(), parent_id.to_string()); - // Only insert if not already present (first child's translation wins) - translations - .entry(parent_key) - .or_insert_with(|| Translation { - ja: None, - ja_hrkt: None, - en: None, - zh: None, - ko: None, - }); - let parent_entry = translations - .get_mut(&("stops".to_string(), parent_id.to_string())) - .unwrap(); - match language { - "ja" if parent_entry.ja.is_none() => { - parent_entry.ja = Some(translation_text.to_string()) - } - "ja-Hrkt" if parent_entry.ja_hrkt.is_none() => { - parent_entry.ja_hrkt = Some(translation_text.to_string()) - } - "en" if parent_entry.en.is_none() => { - parent_entry.en = Some(translation_text.to_string()) - } - "zh-Hans" | "zh-Hant" | "zh" if parent_entry.zh.is_none() => { - parent_entry.zh = Some(translation_text.to_string()) - } - "ko" if parent_entry.ko.is_none() => { - parent_entry.ko = Some(translation_text.to_string()) - } - _ => {} - } + set_translation_language(&mut translations, parent_id, language, text, false); } } + if !field_value.is_empty() { + set_translation_language(&mut translations, field_value, language, text, true); + } } Ok(translations) } +/// Set one language field on the `("stops", key)` translation entry. When +/// `overwrite` is false the value is only filled if currently absent (used for +/// parent-stop aggregation where the first child wins). +fn set_translation_language( + translations: &mut HashMap<(String, String), Translation>, + key: &str, + language: &str, + text: &str, + overwrite: bool, +) { + let entry = translations + .entry(("stops".to_string(), key.to_string())) + .or_default(); + let slot = match language { + "ja" => &mut entry.ja, + "ja-Hrkt" => &mut entry.ja_hrkt, + "en" => &mut entry.en, + "zh-Hans" | "zh-Hant" | "zh" => &mut entry.zh, + "ko" => &mut entry.ko, + _ => return, + }; + if overwrite || slot.is_none() { + *slot = Some(text.to_string()); + } +} + /// Import agencies from agency.txt async fn import_gtfs_agencies( conn: &mut PgConnection, @@ -1393,15 +1405,22 @@ async fn import_gtfs_stops( .filter(|s| !s.is_empty()) .map(|s| s.to_string()); - // Get translations - let key = ("stops".to_string(), original_stop_id.to_string()); - let translation = translations.get(&key); - - let stop_name_k = translation.and_then(|t| t.ja_hrkt.clone()); + // Get translations. Feeds key stop_name translations by either record_id + // (== stop_id) or field_value (== the Japanese stop_name), so try both. + let translation = translations + .get(&("stops".to_string(), original_stop_id.to_string())) + .or_else(|| translations.get(&("stops".to_string(), stop_name.clone()))); + + // Some feeds (Keio, Tokyu community) provide ja-Hrkt readings as + // half-width katakana, so normalize to full-width for the katakana column. + let stop_name_k = translation + .and_then(|t| t.ja_hrkt.clone()) + .map(|k| to_fullwidth_katakana(&k)); // Prefer the official English translation, but fall back to a Hepburn // romanization of the kana reading when the feed ships no `en` row - // (e.g. Keio Bus). This keeps every English-facing surface — station - // name, name search, and romanized route/headsign names — populated. + // (e.g. Tokyu community, or any en-less stop). This keeps every + // English-facing surface — station name, name search, and romanized + // route/headsign names — populated. let stop_name_r = translation .and_then(|t| t.en.clone()) .filter(|s| !s.is_empty()) @@ -3570,6 +3589,67 @@ mod tests { assert_eq!(parse_gtfs_time("aa:bb:cc"), None); } + #[test] + fn test_load_gtfs_translations_field_value_keyed() { + // Keio and the Tokyu community feeds key stop_name translations by + // field_value (the Japanese stop_name) with an empty record_id, ship + // ja-Hrkt as half-width katakana, and use the 7-column layout that + // includes record_sub_id. + let dir = std::env::temp_dir().join("stationapi_test_translations_fv"); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).unwrap(); + std::fs::write( + dir.join("translations.txt"), + "table_name,field_name,language,translation,record_id,record_sub_id,field_value\n\ + stops,stop_name,ja,西八王子駅南口,,,西八王子駅南口\n\ + stops,stop_name,ja-Hrkt,ﾆｼﾊﾁｵｳｼﾞｴｷﾐﾅﾐｸﾞﾁ,,,西八王子駅南口\n\ + stops,stop_name,en,Nishi-hachioji Sta. South,,,西八王子駅南口\n", + ) + .unwrap(); + + let translations = load_gtfs_translations(&dir).unwrap(); + let t = translations + .get(&("stops".to_string(), "西八王子駅南口".to_string())) + .expect("field_value-keyed translation should be found by stop_name"); + assert_eq!(t.ja_hrkt.as_deref(), Some("ﾆｼﾊﾁｵｳｼﾞｴｷﾐﾅﾐｸﾞﾁ")); + assert_eq!(t.en.as_deref(), Some("Nishi-hachioji Sta. South")); + // The half-width reading normalizes to full-width katakana for storage. + assert_eq!( + to_fullwidth_katakana(t.ja_hrkt.as_deref().unwrap()), + "ニシハチオウジエキミナミグチ" + ); + + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn test_load_gtfs_translations_record_id_keyed() { + // Seibu keys translations by record_id (the stop_id, with a "-01" pole + // suffix) in a 6-column layout without record_sub_id, and ships ja-Hrkt + // as full-width katakana. The parent stop_id resolves too. + let dir = std::env::temp_dir().join("stationapi_test_translations_rid"); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).unwrap(); + std::fs::write( + dir.join("translations.txt"), + "table_name,field_name,language,translation,record_id,field_value\n\ + stops,stop_name,ja,西武百貨店前,20001-01,\n\ + stops,stop_name,ja-Hrkt,セイブヒャッカテンマエ,20001-01,\n\ + stops,stop_name,en,Seibu hyakkaten-mae,20001-01,\n", + ) + .unwrap(); + + let translations = load_gtfs_translations(&dir).unwrap(); + let t = translations + .get(&("stops".to_string(), "20001-01".to_string())) + .expect("record_id-keyed translation should be found by stop_id"); + assert_eq!(t.en.as_deref(), Some("Seibu hyakkaten-mae")); + // Parent stop_id (pole suffix dropped) also resolves to the reading. + assert!(translations.contains_key(&("stops".to_string(), "20001".to_string()))); + + let _ = std::fs::remove_dir_all(&dir); + } + #[test] fn test_parse_odpt_time_adds_gtfs_seconds() { assert_eq!(parse_odpt_time("08:30"), Some("08:30:00".to_string()));