makes stripe source to work with modern stripe client

.claude/skills/refresh-source/SKILL.md

@@ -0,0 +1,81 @@
+---
+name: refresh-source
+description: Audit a verified source against the current state of its upstream API/SDK, find breakages and outdated patterns, and propose (or implement) a fix plan with credential-free unit tests. Use when users report a source is outdated or broken, or for periodic source maintenance.
+argument-hint: <source-name> -- [optional hints what to pay attention to]
+---
+
+# Refresh a verified source
+
+Arguments: `$ARGUMENTS` — first token is the source folder name under `sources/`, anything after `--` is user hints (specific user reports, suspected areas).
+
+Work through the phases in order. Do not skip phase 0 — wrong conclusions are usually formed by analyzing code that is no longer alive or pins that no longer reflect reality.
+
+## Phase 0 — Ground truth
+
+1. Read everything in `sources/<name>/`, `sources/<name>_pipeline.py`, `tests/<name>/`.
+2. `git log --oneline -- sources/<name>` — look for past removals/refactors. **Hunt dead code**: files, fixtures, conftest helpers, docstrings and README claims orphaned by earlier PRs. Dead code should be deleted, never "fixed". Dead code may also be the only reason for a heavy dependency — deleting it can drop whole packages from `requirements.txt`.
+3. Compare the THREE dependency declarations and flag mismatches:
+   - `sources/<name>/requirements.txt` — what `dlt init` users actually install (often unbounded!)
+   - `pyproject.toml` dependency group `<name>` — what CI tests
+   - any version pinned in code (e.g. `api_version = "..."`)
+   A CI group pinned years behind an unbounded requirements.txt means CI is testing a different client than users get — that is how breakage ships silently.
+4. Check open/closed GitHub issues for the source (`gh search issues`) and any user hints from the arguments.
+
+## Phase 1 — Research the upstream (web search, authoritative sources only)
+
+Use WebSearch/WebFetch against official docs, SDK changelogs, and SDK migration guides (vendor docs site, vendor GitHub wiki/CHANGELOG). For each finding record the source URL.
+
+- Current SDK major version and every breaking change between the version CI pins and today (migration guides are gold: object model changes, removed dict-inheritance, auth patterns, deprecated module-level clients).
+- Current API version / versioning model; what the code pins vs. what is current; breaking changes in between (renamed/removed/restructured fields the source or its helpers depend on).
+- Official request/response shapes the code touches: list envelopes, pagination cursors, filter parameter formats, retention limits (e.g. event retention windows). Capture documented example payloads — they become test fixtures in phase 4.
+
+## Phase 2 — Reproduce empirically. Never claim a breakage you haven't run.
+
+Use ephemeral environments to prove each suspected breakage and each proposed fix, at **both ends** of the allowed dependency range:
+
+```sh
+cd /tmp && uv run --with "<sdk>==<oldest-allowed>" python - <<'EOF' ... EOF
+cd /tmp && uv run --with "<sdk>==<newest>" python - <<'EOF' ... EOF
+```
+
+Construct real SDK objects (e.g. `construct_from`-style factories) and exercise the exact code path that is suspected broken. Subtle version differences matter (e.g. a conversion helper being shallow in one major and recursive in another) — only execution reveals them.
+
+## Phase 3 — Plan (present before implementing)
+
+Tier the findings; keep tiers separable:
+
+- **P0 — unbreak**: minimal fixes that work across the whole supported dependency range; align CI group with what fresh users install; fix requirements bounds (add upper bound to majors not yet verified). Must NOT change loaded table schemas.
+- **P1 — modernize client**: new SDK client patterns, drop global state, configurable API version. Anything that changes response shapes **changes loaded schemas for existing users** — call this out explicitly as breaking and keep it out of P0.
+- **P2 — features**: new endpoints, incremental improvements.
+
+Prefer dlt built-ins over hand-rolled code (`dlt.common.time.ensure_pendulum_datetime`, `dlt.common` logger, rest_api helpers) — but verify the utility exists in the oldest dlt allowed by the source's requirements.txt before using it.
+
+## Phase 4 — Tests (the biggest pain point)
+
+Existing source tests are usually credential-gated integration tests that never run for contributors. Always add **credential-free unit tests**:
+
+- New `tests/<name>/test_helpers.py` (module-based `def test_*() -> None`, parametrize with readable ids — see `.claude/rules/testing.md`).
+- **Use proven shapes, not invented ones**: fixtures must mirror request/response shapes, pagination envelopes and cursor semantics found in official docs during phase 1 (cite which doc page a fixture mirrors when it isn't obvious).
+- Mock at the transport/SDK boundary with **real SDK objects** (`construct_from` etc.), monkeypatching the API call — so the installed SDK's serialization actually runs instead of a MagicMock lying about it.
+- Cover: pagination control flow (cursor passed from last item, stop on terminal page flag), parameter construction (filters, special-cased endpoints), and any date/type conversion helpers with one case per accepted input type.
+- Run the unit tests against both ends of the SDK range (phase 2 envs) before declaring the range supported.
+
+## Phase 5 — Verify and finish
+
+1. `uv lock` if pyproject changed; ensure resolution works on all supported Python versions.
+2. `pytest tests/<name>/test_helpers.py` — must pass without anything in `sources/.dlt`.
+3. Integration tests `pytest tests/<name>` only if credentials exist; otherwise state plainly they were not run.
+4. `make lint-code` and `make format-lint`.
+5. Update stale docstrings/README touched by the changes. Report findings with file:line references and the doc URLs that back each claim.
+
+## Phase 6 — Assess the public dlt docs
+
+The user-facing docs live in the separate public `dlt-hub/dlt` repo at
+`docs/website/docs/dlt-ecosystem/verified-sources/<name>.md` and are also served to LLMs via
+https://dlthub.com/docs/llms.txt — one fix covers both. Fetch the file
+(`gh api repos/dlt-hub/dlt/contents/docs/...`) and review it against the refreshed source:
+
+- **Code drift**: constant names, default endpoint tuples, function signatures and links quoted in the docs must match `settings.py`/`__init__.py` exactly — docs commonly lag source-repo PRs (e.g. an endpoint moved between ENDPOINTS tuples).
+- **Copy-pasteable examples are the highest-stakes content**: an example demonstrating a removed or unsafe pattern (e.g. incremental loading of an editable endpoint) silently produces wrong data for everyone who pastes it.
+- **Document what the refresh established**: supported SDK version range, pinned upstream API version and its schema implications, accepted input formats, retention limits and other upstream constraints discovered in phase 1.
+- Propose the changes as a diff against the fetched file; the actual PR goes to `dlt-hub/dlt`, not this repo.

.github/workflows/init.yml

@@ -21,7 +21,7 @@ jobs:
       - name: Install uv
         uses: astral-sh/setup-uv@v4
         with:
-          python-version: "3.9"
+          python-version: "3.10"
 
       - name: Install dependencies
         run: uv sync  --only-group dltpure --only-group pytest

.github/workflows/lint.yml

@@ -14,7 +14,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.9", "3.10", "3.11", "3.12"]
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
     runs-on: ubuntu-latest
 
     steps:

.github/workflows/test_destinations_slow.yml

@@ -60,7 +60,7 @@ jobs:
       - name: Install uv
         uses: astral-sh/setup-uv@v4
         with:
-          python-version: "3.9"
+          python-version: "3.10"
 
       - name: Install dependencies
         run: make dev

CONTRIBUTING.md

@@ -120,7 +120,7 @@ ensure you have a working setup.
 
 ### 2. Prepare the development environment
 
-Development on the verified sources repository depends on Python 3.9 or higher and `uv` being
+Development on the verified sources repository depends on Python 3.10 or higher and `uv` being
 available as well as the needed dependencies being installed. Make is used to automate tasks.
 
 1. Install uv: https://docs.astral.sh/uv/getting-started/installation/
@@ -359,13 +359,13 @@ We are happy to add you as contributor to avoid the hurdles of setting up creden
 
 ### Ensuring the correct Python version
 
-Use Python 3.9 for development which is the lowest supported version for `dlt`. You'll need
-`distutils` and `venv`:
+Use Python 3.10 for development which is the lowest supported version for this repository.
+You'll need `distutils` and `venv`:
 
 ```shell
-sudo apt-get install python3.9
-sudo apt-get install python3.9-distutils
-sudo apt install python3.9-venv
+sudo apt-get install python3.10
+sudo apt-get install python3.10-distutils
+sudo apt install python3.10-venv
 ```
 
 `uv` will manage virtual environments for you.

mypy.ini

@@ -1,5 +1,5 @@
 [mypy]
-python_version=3.9
+python_version=3.10
 ignore_missing_imports=false
 strict_optional=false
 warn_redundant_casts=true
@@ -34,6 +34,11 @@ ignore_missing_imports=true
 [mypy-langchain.*]
 ignore_missing_imports=true
 
+# langchain is not installed on python 3.13+ (requires numpy<2), its types resolve
+# to Any there and ignores that are needed on older pythons appear unused
+[mypy-unstructured_data.*]
+warn_unused_ignores=false
+
 [mypy-pydispatch.*]
 ignore_missing_imports=true
 

pyproject.toml

@@ -3,7 +3,7 @@ name = "dlt-verified-sources"
 version = "0.1.0"
 description = "Initial set of dlt sources with demo pipelines installable with `dlt init` command"
 authors = [{ name = "Various Authors", email = "team@dlthub.com" }]
-requires-python = ">=3.9"
+requires-python = ">=3.10, <3.15"
 readme = "README.md"
 license = "MIT"
 dependencies = [
@@ -40,12 +40,14 @@ dev = [
     "pytest-forked>=1.6.0,<2",
     "pendulum>=3.0.0,<4",
     "psycopg2-binary>=2.9.9",
+    # numpy 1.x has no python 3.13 wheels, fork the resolution so 3.13+ gets numpy 2.x
+    # (1.x stays for python < 3.13 where langchain 0.0.x requires numpy<2)
+    "numpy>=2 ; python_version >= '3.13'",
 ]
 sql_database = [
     "sqlalchemy>=1.4",
     "pymysql>=1.0.3,<2",
-    'connectorx>=0.3.3; python_version >= "3.9"',
-    'connectorx>=0.4.0; python_version >= "3.10"',
+    "connectorx>=0.4.0",
 ]
 pg_replication = ["psycopg2-binary>=2.9.9"]
 google_sheets = ["google-api-python-client>=2.78.0,<3"]
@@ -56,9 +58,7 @@ google_analytics = [
     "requests-oauthlib>=1.3.1,<2",
 ]
 stripe_analytics = [
-    "pandas>=2.0.0,<3",
-    "stripe>=5.0.0,<6",
-    "types-stripe>=3.5.2.14,<4",
+    "stripe>=15,<16",
 ]
 asana_dlt = ["asana>=3.2.1,<4"]
 facebook_ads = ["facebook-business>=17.0.2,<18"]
@@ -67,16 +67,17 @@ google_ads = [
     "google-api-python-client>=2.129.0,<3",
 ]
 salesforce = ["simple-salesforce>=1.12.4,<2"]
+# langchain 0.0.x requires numpy<2 which does not build on python 3.13+
 unstructured_data_lint = [
-    "langchain>=0.0.219,<0.0.220",
-    "openai>=0.27.8,<0.28",
+    "langchain>=0.0.219,<0.0.220 ; python_version < '3.13'",
+    "openai>=0.27.8,<0.28 ; python_version < '3.13'",
 ]
 unstructured_data = [
-    "langchain>=0.0.219,<0.0.220",
-    "unstructured>=0.7.10,<0.8",
-    "openai>=0.27.8,<0.28",
-    "chromadb>=0.3.26,<0.4",
-    "tiktoken>=0.4.0,<0.5",
+    "langchain>=0.0.219,<0.0.220 ; python_version < '3.13'",
+    "unstructured>=0.7.10,<0.8 ; python_version < '3.13'",
+    "openai>=0.27.8,<0.28 ; python_version < '3.13'",
+    "chromadb>=0.3.26,<0.4 ; python_version < '3.13'",
+    "tiktoken>=0.4.0,<0.5 ; python_version < '3.13'",
 ]
 mongodb = [
     "pymongo>=4.3.3,<5",

sources/stripe_analytics/__init__.py

@@ -1,4 +1,4 @@
-""" This source uses Stripe API and dlt to load data such as Customer, Subscription, Event etc. to the database and to calculate the MRR and churn rate. """
+""" This source uses Stripe API and dlt to load data such as Customer, Subscription, Event etc. to the database. """
 
 from typing import Any, Dict, Generator, Iterable, Optional, Tuple
 

sources/stripe_analytics/helpers.py

@@ -3,7 +3,7 @@
 from typing import Any, Dict, Iterable, Optional, Union
 
 import stripe
-from dlt.common import pendulum
+from dlt.common.time import ensure_pendulum_datetime
 from dlt.common.typing import TDataItem
 from pendulum import DateTime
 
@@ -40,12 +40,8 @@ def pagination(
 
 
 def transform_date(date: Union[str, DateTime, int]) -> int:
-    if isinstance(date, str):
-        date = pendulum.from_format(date, "%Y-%m-%dT%H:%M:%SZ")
-    if isinstance(date, DateTime):
-        # convert to unix timestamp
-        date = int(date.timestamp())
-    return date
+    # convert ISO 8601 strings, datetimes and unix timestamps to a unix timestamp
+    return int(ensure_pendulum_datetime(date).timestamp())
 
 
 def stripe_get_data(
@@ -65,4 +61,6 @@ def stripe_get_data(
     resource_dict = getattr(stripe, resource).list(
         created={"gte": start_date, "lt": end_date}, limit=100, **kwargs
     )
-    return dict(resource_dict)
+    # to_dict works across stripe-python versions, dict() conversion broke in v15
+    # when StripeObject stopped inheriting from dict
+    return resource_dict.to_dict()  # type: ignore[no-any-return]

sources/stripe_analytics/requirements.txt

@@ -1,3 +1,2 @@
-pandas>=2.0.0
-stripe>=5.0.0
+stripe>=5.0.0,<16
 dlt>=0.5.1

sources/unstructured_data/async_index.py

@@ -27,7 +27,7 @@ async def aquery(
         chain = RetrievalQA.from_chain_type(
             llm, retriever=self.vectorstore.as_retriever(), **kwargs
         )
-        return await chain.arun(question)
+        return str(await chain.arun(question))
 
 
 class AVectorstoreIndexCreator(VectorstoreIndexCreator):