Releases: datalad/datalad-next
Release list
1.6.0 (2026-05-17)
💫 New features
iter_gitdiff:- add support for pathspec-constrained reporting [76d8db7f]
- atomic:
- describe atomicity property [f07a613b]
- atomic upload for
url_operations.file[4a8aa14f] - atomic upload for url_operations.ssh [116ac16f]
- delete:
- add delete to HttpUrlOperations [dfbf1e50]
- add delete to
SshUrlOperations[fb4aeec8]
🐛 Bug Fixes
- guard handle_401 against IndexError when token meets Basic-only server [d947636b]
- include ZipArchiveOperations in Archivist's handlers [54d783f8]
- proper import of version information to the top-level module [ed1ca140]
- delay import to have import subject be impacted by patches [5ee1a7b8]
CreateSiblingRia:- canonify push-url correctly [4f1f578b]
decode_bytes:- error handling led to data loss in subsequent chunks [dfa1d20b]
- handle multiple errors [7a73d8ce]
- always backslashreplace when asked to [0ac7974a]
gitworktree:- decode bytes before itemization [5fe82ead]
iter_gitdiff:- adjust conditional acknowledging that
GitTreeItem.nameis astr[21db7511] - enable multi-level submodule recursion [94bd3c50]
- yield copy of container item when it is also further processed [78d71e82]
- correct
git diff-index|treecommand for submodule recursion [c37dd315] - corrected type annotation in internal helper [0eb4ffb4]
- decode bytes before itemization [8b188142]
- adjust conditional acknowledging that
iter_gittree:- decode bytes before itemization [c8598545]
- delete:
- unify semantics of delete-operations [80bc38c3]
- set write-permission in
SshUrlOperations.delete[86c728fd]
- tests:
- bring back datalad core wtf-test [6d2d44a5]
- remove print-statement from test [67acb453]
- remove align_pattern performance test [3642879a]
- types:
- add type for
prop[f96dd3cf] - fix exception arguments [7fa458d3]
- fix type annotations in alig_pattern [247ee261]
- add type for
- uncurl:
- add close-handler to uncurl [c9b2a4c8]
🏠 Refactorings
- import
PRE_INIT_COMMIT_SHAfromdatalad_core[43c72541] - homogenize deprecation pattern for
datalad_next.iterable_subprocess[cc846f5b] - homogenize deprecation pattern for
datalad_next.iterools[7bd83268] - use
gitpathspecmodule fromdatasaladlibrary [337187ff] iter_gitdiff:- establish new top-level internal helper [f922c7bd]
- new dedicated helper to yield from a submodule [fbb0318e]
- discontinue use of
kwargsdict [232dba4b] - flip conditional to simplify internal helper [39505865]
- dedicated function for Git output ->
GitDiffItem[a513b8c8] - dissolve kwarg-dict to improve type annotation [b88e21f3]
iter_gitstatus:- reduce variable recycling to avoid confusing type-checker [2aa77052]
iterable_subprocess:- removed package and use from
datasalad[4ee9c272]
- removed package and use from
itertools:- removed package and use from
datasalad[a2a992d5]
- removed package and use from
📝 Documentation
- add maintainability badge [abd0e4b7]
- add
datalad_next.shell.shell()to docs [e49ca134] - Add official Hatch badge to README.md [18fa8c63]
- trim module API documentation to an overview table of content [704aeedf]
- (temporarily) disable the inclusion of manpages in the docs [e3e588af]
- remove unused intersphinx extension [761f232f]
- correct spelling in doc-strings [6511cbd9]
- exclude memory addresses from documentation [0d2440de]
iter_gitdiff:- add docstring for
eval_submodule_stateargument [e786921b]
- add docstring for
- Contributing guide:
- cheat sheet on
hatch[8f2f7d69]
- cheat sheet on
🛡 Tests
- cover token auth with Basic-only challenge [1e33929d]
- enable zip testing [2d2b1526]
- simplify hatch test setup [e8bc0268]
- fix test_tree_with_broken_symlinks on Python 3.13 [25ac6ef0]
- fix ruff setup (used wrong path) [865665ac]
- disable test with erratic behavior after hatch-based execution [1c3f3130]
- disable test_wtf of datalad-core-4 [4477de88]
iter_gitdiff:- adjust type-annotation to match reality [5cb96362]
- new test to show that recursion is broken [858036fa]
- type-annotate internal helper [b4bec180]
test_gitdiff:- disable kwargs type-checking [04d56150]
1.5.0 (2024-06-16)
💫 New features
- new support subpackage for Git's pathspecs [8d05f1cd]
- add pathspec support to
iter_(gitworktree|submodules)()[7622267c] GitWorktreeFileSystemItem:- method to convert
GitWorktreeItem[30858780]
- method to convert
iter_gitworktree:- added submodule recursion with pathspec support [2851ec4c]
- add ability to report untracked content only [2763b10f]
iter_submodules:- add
match_containingmode [8025b7bd]
- add
🐛 Bug Fixes
iter_gitstatus:- support newly added submodules [fd197ad6]
iter_gitstatus:- rectify usage of unsupported
untracked=nomode in tests [673e6c22]
- rectify usage of unsupported
- Patches for DataLad (core):
- fix
create_sibling_riapatching [78e29b5d]
- fix
📝 Documentation
- Contributing guide:
- expand description of expected commit messages [04f68b49]
1.4.1 (2024-05-22)
1.4.0 (2024-05-17)
🐛 Bug Fixes
-
RIA over SSH access from Mac clients to Linux server was broken
due to an inappropriate platform check that assumed that local and
remote platform are identical.
Fixes datalad/datalad#7536 via #653 (by @mih) -
next-statushas received a number of fixes:- It no longer issues undesirable modification reports
that are based onmtimechanges alone (i.e., no content change).
Fixes #639 via #650 (by @mih) - It now detects staged changes in repositories with no
commit.
Fixes #680 via #681 (by @mih) next-status -r mononow reports on new commits in submodules.
Previously this was ignored, leading to the impression of
clean datasets despite unsaved changes.
Fixes #645 via #679 (by @mih)
- It no longer issues undesirable modification reports
-
iter_annexworktree()can now also be used on plain Git repos,
and would behave exactly as if reporting on non-annexed files
in a git-annex repo. Previously, a crypticiterable did not yield matching item for route-in item, cardinality mismatch?error was
issued in this case.
Fixes #670 via #673 (by @mih)
💫 Enhancements and new features
-
datalad_next.shellprovides a context manager for (long-running)
shell or interpreter subprocesses. Within the context any number of
commands can be executed in such a shell, and each command can
process input (iterables), and yield output (iterables). This feature
is suitable for running and controlling "remote shells" like a login
shell on a server via SSH. A range of utilities is provided to
employ this functionality for special purpose implementations
(e.g., accept fixed-length or variable-length process output).
A suite of operations like download/upload file to a remote shell is
provided for POSIX-compliant shellsdatalad_next.shell.operations.posix.
#596 (by @christian-monch) -
A rewrite of
SSHRemoteIO, the RIA SSH-operations implementation from
datalad-core is provided as a patch. It is based on the newshell
feature, and provides more robust operations. It's IO performance is
at the same level asscp-based down/uploads. In contrast to the
original implementation, it support fine-grained progress reporting
for uploads and downloads.
Via #655 (by @mih) -
The
SpecialRemotebase class in datalad-core is patched to support
a standardclose()method for implementing resource release and cleanup
operations. The main special remote entry point has been altered to
run implementations within aclosing()context manager to guarantee
execution of such handlers.
Via #655 (by @mih) -
A new
has_initialized_annex()helper function is provided to
test for a locally initialized annex in a repo.
Via #673 (by @mih) -
iter_annexworktree()can now also be used on plain Git repositories,
and it yields the same output and behavior as running on a git-annex
repository with no annex'ed content (just tracked with Git).
Fixes #670 via #673 (by @mih) -
next-statusanditer_gitstatus()have been improved to
report on further modifications after a file addition has been
originally staged.
Fixes #637 via #679 (by @mih) -
next-statusresult rendering has been updated to be more markedly
different than git-status's. Coloring is now exclusively
determined by the nature of a change, rather than being partially
similar to git-status's index-updated annotation. This reduces
the chance for misinterpretations, and does not create an undesirable
focus on the Git index (which is largely ignored by DataLad).
Fixes #640 via #679 (by @mih) -
A large 3k-line patch set replaces almost the entire RIA implementation,
including the ORA special remote, and thecreate-sibling-riacommand.
The new implementation brings uniform support for Windows clients, progress
reporting for uploads and downloads via SSH, and a faster and more
robust behavior for SSH-based operations (based on the new remote
shell feature).
Fixes #654 via #669 (by @christian-monch)
📝 Documentation
- Git-related subprocess execution helpers are now accessible in the
rendered documentation, and all supported file collections are now
mentioned in thels-file-collectioncommand help.
Fixes #668 via #671 (by @mih)
🛡 Tests
- Test setup has been improved to support a uniform, datalad-next
enabled environment for subprocesses too. This extends the scope
of testing to special remote implementations and other code that
is executed in subprocesses, and relies on runtime patches.
See https://github.com/datalad/datalad-next/pull/i665 (by @mih)
1.3.0 (2024-03-19)
💫 Enhancements and new features
-
Code organization is adjusted to clearly indicate what is part of the
package's public Python API. Anything that can be imported directly from
the top-level of any sub-package is part of the public API.
As an example:from datalad_next.runners import iter_git_subproc
imports a part of the public API, but
from datalad_next.runners.git import iter_git_subprocdoes not.
SeeREADME.mdfor more information.
Fixes #613 via #615 (by @mih), #617 (by @mih), #618 (by @mih), #619 (by @mih),
#620 (by @mih), #621 (by @mih), #622 (by @mih), #623 (by @mih) -
New
patched_envcontext manager for patching a process'
environment. This avoids the for importingunittestoutside
test implementations.
Via #633 (by @mih) -
call_git...()functions received a newforce_c_locale
parameter. This can be set whenever Git output needs to be parsed
to force running the command withLC_ALL=C. Such an environment
manipulation is off by default and not done unconditionally to
let localized messaging through in a user's normal locale.
🐛 Bug Fixes
datalad-annex::Git remote helper now tests for a repository
deposit, and distinguishes an absent remote repository deposit
vs cloning from an empty repository deposit. This rectifies
confusing behavior (successful clones of empty repositories
from broken URLs), but also fixes handling of subdataset clone
candidate handling inget(which failed to skip inaccessible
datalad-annex::URLs for the same reason).
Fixes #636 via
#638 (by @mih)
📝 Documentation
- API docs have been updated to include all top-level symbols
of any sub-package, or in other words: the public API.
See #627 (by @mih)
🏠 Internal
-
The
treecommand no longer uses thesubdatasetscommand
for queries, but employs the recently introducediter_submodules()
for leaner operations.
See #628 (by @mih) -
call_git...()functions are established as the only used abstraction
to interface with Git and git-annex commands outside the use in
DataLad'sRepoclasses. Any usage of DataLad's traditional
Runnerfunctionality is discontinued.
Fixes #541 via
#632 (by @mih) -
Type annotations have been added to the implementation of the
uncurlgit-annex remote. A number of unhandled conditions have
been discovered and were rectified.
1.2.0 (2024-02-02)
🐛 Bug Fixes
- Fix an invalid escape sequence in a regex that caused a syntax warning.
Fixes #602 via
#603 (by @mih)
💫 Enhancements and new features
- Speed up of status reports for repositories with many submodules.
An early presence check for submodules skips unnecessary evaluation
steps. Fixes #606 via
#607 (by @mih)
🏠 Internal
- Fix implementation error in
ParamDictatorclass that caused a test
failure. The class itself is unused and has been scheduled for removal.
See #611 and
#610 (by @christian-monch)
🛡 Tests
-
Promote a previously internal fixture to provide a standard
modified_datasetfixture. This fixture is sessions-scope, and
yields a dataset with many facets of modification, suitable for
testing change reporting. The fixture verifies that no
modifications have been applied to the testbed. (by @mih) -
iterable_subprocesstests have been robustified to better handle the
observed diversity of execution environments. This addresseses, for example,
https://bugs.debian.org/1061739.
#614 (by @christian-monch)
1.1.0 -- Iterate!
💫 Enhancements and new features
-
A new paradigm for subprocess execution is introduced. The main
workhorse isdatalad_next.runners.iter_subproc. This is a
context manager that feeds input to subprocesses via iterables,
and also exposes their output as an iterable. The implementation
is based on https://github.com/uktrade/iterable-subprocess, and
a copy of it is now included in the sources. It has been modified
to work homogeneously on the Windows platform too.
This new implementation is leaner and more performant. Benchmarks
suggest that the execution of multi-step pipe connections of Git
and git-annex commands is within 5% of the runtime of their direct
shell-execution equivalent (outside Python).
See #538 (by @mih),
#547 (by @mih).With this change a number of additional features have been added,
and internal improvements have been made. For example, any
use ofThreadedRunnerhas been discontinued. See
#539 (by @christian-monch),
#545 (by @christian-monch),
#550 (by @christian-monch),
#573 (by @christian-monch)-
A new
itertoolsmodule was added. It provides implementations
of iterators that can be used in conjunction withiter_subproc
for standard tasks. This includes the itemization of output
(e.g., line-by-line) across chunks of bytes read from a process
(itemize), output decoding (decode_bytes), JSON-loading
(json_load), and helpers to construct more complex data flows
(route_out,route_in). -
The
more_itertoolspackage has been added as a new dependency.
It is used fordatalad-nextiterator implementations, but is also
ideal for client code that employed this new functionality. -
A new
iter_annexworktree()provides the analog ofiter_gitworktree()
for git-annex repositories. -
iter_gitworktree()has been reimplemented arounditer_subproc. The
performance is substantially improved. -
iter_gitworktree()now also provides file pointers to
symlinked content. Fixes #553
via #555 (by @mih) -
iter_gitworktree()anditer_annexworktree()now support single
directory (i.e., non-recursive) reporting too.
See #552 -
A new
iter_gittree()that wrapsgit ls-treefor iterating over
the content of a Git tree-ish.
#580 (by @mih). -
A new
iter_gitdiff()wrapsgit diff-tree|filesand provides a flexible
basis for iteration over changesets.
-
-
PathBasedItem, a dataclass that is the bases for many item types yielded
by iterators now more strictly separatesnameproperty from path semantics.
The name is a plain string, and an additional, explicitpathproperty
provides it in the form of aPath. This simplifies code (the
_ZipFileDirPathutility class became obsolete and was removed), and
improve performance.
Fixes #554 and
#581 via
#583 (by @mih) -
A collection of helpers for running Git command has been added at
datalad_next.runners.git. Direct uses of datalad-core runners,
orsubprocess.run()for this purpose have been replaced with call
to these utilities.
#585 (by @mih) -
The performance of
iter_gitworktree()has been improved by about
10%. Fixes #540
via #544 (by @mih). -
New
EnsureHashAlgorithmconstraint to automatically expose
and verify algorithm labels fromhashlib.algorithms_guaranteed
Fixes #346 via
#492 (by @mslw @adswa) -
The
archivistremote now supports archive type detection
from*E-type annex keys for.tgzarchives too.
Fixes #517 via
#518 (by @mih) -
iter_zip()uses a dedicated, internalPurePathvariant to report on
directories (_ZipFileDirPath). This enables more straightforward
item.name in zip_archivetests, which require a trailing/for
directory-type archive members.
#430 (by @christian-monch) -
A new
ZipArchiveOperationsclass added support for ZIP files, and enables
their use together with thearchivistgit-annex special remote.
#578 (by @christian-monch) -
datalad ls-file-collectionhas learned additional collections types:-
The new
zipfilecollection type that enables uniform reporting on
the additional archive type. -
The new
annexworktreecollection that enhances thegitworktree
collection by also reporting on annexed content, using the new
iter_annexworktree()implementation. It is about 15% faster than a
datalad --annex basic --untracked no -e no -t eval. -
The new
gittreecollection for listing any Git tree-ish. -
A new
iter_gitstatus()can replace the functionality of
GitRepo.diffstatus()with a substantially faster implementation.
It also provides a novelmonorecursion mode that completely
hides the notion of submodules and presents deeply nested
hierarchies of datasets as a single "monorepo".
#592 (by @mih)
-
-
A new
next-statuscommand provides a substantially faster
alternative to the datalad-corestatuscommand. It is closely
aligned togit statussemantics, only reports changes (not repository
listings), and supports type change detection. Moreover, it exposes
the "monorepo" recursion mode, and single-directory reporting options
ofiter_gitstatus(). It is the first command to usedataclass
instances as result types, rather than the traditional dictionaries. -
SshUrlOperationsnow supports non-standard SSH ports, non-default
user names, and custom identity file specifications.
Fixed #571 via
#570 (by @mih) -
A new
EnsureRemoteNameconstraint improves the parameter validation
ofcreate-sibling-webdav. Moreover, the command has been uplifted
to support uniform parameter validation also for the Python API.
Missing required remotes, or naming conflicts are now detected and
reported immediately before the actual command implementation runs.
Fixes #193 via
#577 (by @mih) -
datalad_next.repo_utilsprovide a collection of implementations
for common operations on Git repositories. Unlike the datalad-core
Repoclasses, these implementations do no require a specific
data structure or object type beyond aPath.
🐛 Bug Fixes
-
Add patch to fix
update's target detection for adjusted mode datasets
that can crash under some circumstances.
See datalad/datalad#7507, fixed via
#509 (by @mih) -
Comparison with
isand a literal was replaced with a proper construct.
While having no functional impact, it removes an uglySyntaxWarning.
Fixed #526 via
#527 (by @mih)
📝 Documentation
- The API documentation has been substantially extended. More already
documented API components are now actually renderer, and more documentation
has been written.
🏠 Internal
-
Type annotations have been extended. The development workflows now inform
about type annotation issues for each proposed change. -
Constants have been migrated to
datalad_next.consts.
#575 (by @mih)
🛡 Tests
-
A new test verifies compatibility with HTTP serves that do not report
download progress.
#369 (by @christian-monch) -
The overall noise-level in the test battery output has been reduced
substantially. INFO log messages are no longer shown, and command result
rendering is largely suppressed. New test fixtures make it easier
to maintain tidier output:reduce_logging,no_result_rendering.
The contribution guide has been adjusted encourage their use. -
Tests that require an unprivileged system account to run are now skipped
when executed as root. This fixes an issue of the Debian package.
#593 (by @adswa)
Full Changelog: 1.0.2...1.1.0
1.0.2 -- Debianize!
🏠 Internal
- The
www-authenticatedependencies is dropped. The functionality is
replaced by arequests-based implementation of an alternative parser.
This trims the dependency footprint and facilitates Debian-packaging.
The previous test cases are kept and further extended.
Fixes #493 via
#495 (by @mih)
🛡 Tests
- The test battery now honors the
DATALAD_TESTS_NONETWORKenvironment
variable and downgrades by skipping any tests that require external
network access. (by @mih)
1.0.1 -- Fix me some 3.8
🐛 Bug Fixes
-
Fix f-string syntax in error message of the
uncurlremote.
#455 (by @christian-monch) -
FileSystemItem.from_path()now honors itslink_targetparameter, and
resolves a target for any symlink item conditional on this setting.
Previously, a symlink target was always resolved.
Fixes #462 via
#464 (by @mih) -
Update the vendor installation of versioneer to v0.29. This
resolves an installation failure with Python 3.12 due to
the removal of an ancient class.
Fixes #475 via
#483 (by @mih) -
Bump dependency on Python to 3.8. This is presently the oldest version
still supported upstream. However, some functionality already used
3.8 features, so this is also a bug fix.
Fixes #481 via
#486 (by @mih)
💫 Enhancements and new features
- Patch datalad-core's
runcommand to honor configuration defaults
for substitutions. This enables placeholders like{python}that
point tosys.executableby default, and need not be explicitly
defined in system/user/dataset configuration.
Fixes #478 via
#485 (by @mih)
📝 Documentation
-
Include
gitworktreeamong the available file collection types
listed inls-file-collection's docstring. Fixes
#470 via
#471 (by @mslw) -
The renderer API documentation now includes an entrypoint for the
runner-related functionality and documentation at
https://docs.datalad.org/projects/next/en/latest/generated/datalad_next.runners.html
Fixes #466 via
#467 (by @mih)
🛡 Tests
1.0.0 -- semantic versioning from here
This release represents a milestone in the development of the extension.
The package is reorganized to be a collection of more self-contained
mini-packages, each with its own set of tests.
Developer documentation and guidelines have been added to aid further
development. One particular goal is to establish datalad-next as a proxy
for importing datalad-core functionality for other extensions. Direct imports
from datalad-core can be minimized in favor of imports from datalad-next.
This helps identifying functionality needed outside the core package,
and guides efforts for future improvements.
The 1.0 release marks the switch to a more standard approach to semantic
versioning. However, although a substantial improvements have been made,
the 1.0 version nohow indicates a slowdown of development or a change in the
likelihood of (breaking) changes. They will merely become more easily
discoverable from the version label alone.
Notable high-level features introduced by this major release are:
-
The new
UrlOperationsframework to provide a set of basic operations like
download,upload,statfor different protocols. This framework can be
thought of as a replacement for the "downloaders" functionality in
datalad-core -- although the feature list is not 100% overlapping. This new
framework is more easily extensible by 3rd-party code. -
The
Constraintsframework elevates parameter/input validation to the next
level. In contrast to datalad-core, declarative input validation is no longer
limited to the CLI. Instead, command parameters can now be validated regardless
of the entrypoint through which a command is used. They can be validated
individually, but also sets of parameters can be validated jointly to implement
particular interaction checks. All parameter validations can now be performed
exhaustive, to present a user with a complete list of validation errors, rather
then the fail-on-first-error method implemented exclusively in datalad-core.
Validation errors are now reported using dedicated structured data type to aid
their communication via non-console interfaces. -
The
Credentialssystem has been further refined with more homogenized
workflows and deeper integration into other subsystems. This release merely
represents a snapshot of continued development towards a standardization of
credential handling workflows. -
The annex remotes
uncurlandarchivistare replacements for the
datalad-core implementationsdataladanddatalad-archive. The offer
substantially improved configurability and leaner operation -- built on the
UrlOperationsframework. -
A growing collection of iterator (see
iter_collections) aims to provide
fast (and more Pythonic) operations on common data structures (Git worktrees,
directories, archives). The can be used as an alternative to the traditional
Repoclasses (GitRepo,AnnexRepo) from datalad-core. -
Analog to
UrlOperationstheArchiveOperationsframework aims to provide
an abstraction for operations on different archive types (e.g., TAR). The
represent an alternative to the traditional implementations of
ExtractedArchiveandArchivesCachefrom datalad-core, and aim at leaner
resource footprints. -
The collection of runtime patches for datalad-core has been further expanded.
All patches are now individually documented, and applied using a set of standard
helpers (see http://docs.datalad.org/projects/next/en/latest/patches.html).
For details, please see the changelogs of the 1.0.0 beta releases below.
💫 Enhancements and new features
TarArchiveOperationsis the first implementation of theArchiveOperations
abstraction, providing archive handlers with a set of standard operations:
🐛 Bug Fixes
-
Make
TarfileItem.namebe of typePurePosixPathto reflect the fact
that a TAR archive can contain members with names that cannot be represent
unmodified on a non-POSIX file system.
#422 (by @mih)
An analog change is done forZipfileItem.name.
#409 (by @christian-monch) -
Fix
git ls-fileparsing initer_gitworktree()to be compatible with
file names that start with atabcharacter.
#421 (by @christian-monch)
📝 Documentation
-
Expanded guidelines on test implementations.
-
Add missing and fix wrong docstrings for HTTP/WebDAV server related fixtures.
#445 (by @adswa)