Skip to content

Add animated streamlines for vector fields#194

Open
Zarnosch wants to merge 8 commits into
d70-t:mainfrom
Zarnosch:agent/vector-field-streamlines
Open

Add animated streamlines for vector fields#194
Zarnosch wants to merge 8 commits into
d70-t:mainfrom
Zarnosch:agent/vector-field-streamlines

Conversation

@Zarnosch

@Zarnosch Zarnosch commented Jul 23, 2026

Copy link
Copy Markdown

Adds optional animated streamlines for vector datasets.

Gridlook detects u/v, ua/va, and uas/vas pairs automatically. Other vector components, such as ocean currents, can be selected manually in the Flow panel.

The streamlines use cached RK4/3 trajectories with fading, speed-dependent traces. They work across the supported grid types, including HEALPix, and are disabled by default because building the initial trajectory cache can take a moment.

Tested with the ICON HEALPix uas/vas dataset. Unit tests, lint, typecheck, and the production build pass.

Screencast.from.2026-07-23.03-16-20.webm

@Karinon

Karinon commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

First of all really nice feature! Seems to be a really productive session inside the camper van ;). In order to keep my sanity with all those PRs touching dozens of files (especially every single grid-implementation) I would like to merge this one after we merged the ellipsoidal healpix-stuff (and after I checked the code as far as I can). So you probably have to do some rebasing later on..

I am not sure how much we/you can do about this, but when moving the projection center, we get artifacts because the steamlines from the most right position try to reconnect with the streamlines on the most left and do it by creating lines all over the grid. Here is a screenshot:

gridlook

This is a constant problem with all features on the globe..had this issues with the coastlines as well. If we can't resolve it, we should probably disable this feature for funky projection centers, however I think it would be really nice if they would work at least for the azimuthal hybrid projection with Antarctica in the center (in order to project it on the hyperglobe).

A few smaller nitpicks on the UX:

  • Could you add a title to the button when it is disabled? Or even when it is enabled. Having a button-title (I mean like <button title="this is a title"> bla </button>) add automatic tooltips when hovering over the button. Like this we could add an explanation why the button becomes unclickable for certain datasets so that the user is hopefully less confused (this is my most important point).
  • when changing datasets with streamlines enabled the button remains enabled even when the new dataset has not the requirements for streamlines. When I click it again it gets disabled and unclickable. I think I would prefer if the button disables itself when changing datasets and the new dataset does not fulfill the requirements
  • I am not sure if I like having Flow as its own subsection. As of now we are quite "economical" on new subsections and the flow-stuff could fit into the Masks & Layers. I would like to discuss this with other people before doing something.
  • I am also not sure about the checkbox. I deliberately got rid of every single checkbox in the past because I wanted a more "app-like" experience and checkboxes automatically look like some kind of form. But I understand that it would probably look weird with two buttons where one depends on the other. Leave it as it is for now, I would also like to discuss this with other people, probably it is fine as it is.

@Karinon

Karinon commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Btw. I tried it with @leifdenby's data, e.g. this one: https://harmonie-zarr.s3.amazonaws.com/dini/control/2026-07-23T030000Z/single_levels.zarr

Which should have u10 and v10. Why doesn't it work for those? Another thing here is, that Leif Denby's Fork creates an artificial variable wind speed (https://dmidev.org/dini/#::timeindex=6::varname=wind%20speed%20(u10m%20v10m)) which is probably something we want at some point, but I saw already other gridlook-forks with artificial variables and this is probably something we should tackle in a separate PR.

@d70-t

d70-t commented Jul 23, 2026

Copy link
Copy Markdown
Owner

This is a constant problem with all features on the globe..had this issues with the coastlines as well.

I think for my initial azimutal branch, I got it somewhat solved by not projecting on a flat 2d surface, but on a very shallow cone, pointing towards the camera instead. This brings those lines to the back, and if the cone is filled with black, you don't see the lines.
Also, if the camera is parallel (in stead of perspective), the cone will be perfectly hidden.

@Zarnosch

Copy link
Copy Markdown
Author
  • For sure I can put a tooltip in this. Also the automatic deactivation if no vectorfield variables are found. This saying its currently just looking for v/u, ua,va or uas,vas variable name pairs, thats also why u10 and v10 were not detected. Its easy to add that pair also to the automatic vectorfield discovery, however I wanted to also enable users to choose their own u and v vectorfield components, especially if they want to show ocean currents.

  • I also thought if it fits in the Layering and Masks Section but I wanted to keep it as simple as possible for a first draft. I have no hard feeling where this should be located and I strongly appreciate the app like usability.

  • Regarding the artificial variable windspeed, this would make perfect sense to provide this. I already compute it as its effecting, how fast the particles move. However I would like to have the Option for many more artificial variables, for example divergence, curl, or other scalar and vectorfield operations.

  • In the long run I would also like to check if volume rendering of 3d variables could be beneficial. Isolines and Isosurfaces might also be a cool feature. This will most likely not fir in an app like UI and one might hide this behind an Advanced Switch or something like that.
    I am also happy to further discuss this in person.

@Karinon

Karinon commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Ok, I tried it again right now, more points about the UX:

  • What made me more confused with the DMI-Dataset was that the Enable streamlines-Button was disabled by default (because it obviously did not detect the right variables). Seeing that, I completely ignored that I could uncheck the checkbox to select something by myself. Maybe it would make sense to merge the checkbox and the button into one button Enable streamlines. When turned on, you automatically see the two dropdowns which are automatically populated, if we can autodetect variables. Otherwise the user has to fill them by hand
  • It would make sense to enable this stuff via the URL as well

@Zarnosch
Zarnosch marked this pull request as draft July 23, 2026 08:36
@Zarnosch
Zarnosch marked this pull request as ready for review July 23, 2026 09:05
@Zarnosch

Copy link
Copy Markdown
Author

The latest commits should cover all of the discussed points above

Comment thread src/lib/data/ZarrDataManager.ts Outdated
@Karinon

Karinon commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

I have a Gaussian Reduced dataset (locally unfortunetaly, @wachsylon where is the online source of ifs-amip-tco1279.hist.v20240901.atmos.native.2D_monthly/zarr ?)

Looks like this:

gridlook

The color underneath is the U. Something is funky here between Lon -90 and 0. Not sure if this is the data or the implementation. Would be helpful to have the data available.

@wachsylon

Copy link
Copy Markdown
Collaborator

@Karinon

Karinon commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

I asked Codex about the issue with the missing streamlines between -90 and 0:

The local zarr data is complete. For time index 0, avg_10u and avg_10v both have 654400/654400 finite values. The dataset stores longitude as 0..359.777, so the displayed -90..0 region corresponds to raw 270..360, and that band also has finite wind data.
The bug was in Gridlook’s irregular streamline sampler: it indexed source longitudes exactly as stored, but resampled global fields on a -180..179 grid. For this dataset, geokdbush then returned bad/far neighbors for much of -90..0, producing mostly NaN stream samples. Before the fix, the resampled -90..0 band had only 1297/16110 finite samples; after the fix it has 16110/16110.
I fixed it by normalizing global irregular-vector source longitudes into [-180, 180) before building the spatial index in vectorField.ts (line 208) and applying that normalized set in the IrregularVectorField constructor at vectorField.ts (line 714). I also added a regression test for global unstructured fields stored as 0..360 longitudes in vectorField.test.ts (line 102).
Verified with:

npx vitest run tests/unit/lib/data/vectorField.test.ts
npx eslint src/lib/data/vectorField.ts tests/unit/lib/data/vectorField.test.ts --max-warnings 0
git diff --check

Here is the code it created. Please don't add the thousandth Version of "normalizeLongitude". They are everywhere in Gridlook....

// vectorField.ts

function normalizeLongitude180(longitude: number) {
  return ((((longitude + 180) % 360) + 360) % 360) - 180;
}

function normalizeLongitudes180(longitudes: Float32Array) {
  return Float32Array.from(longitudes, normalizeLongitude180);
}

function normalizeGlobalLongitudes(longitudes: Float32Array) {
  const bounds = finiteBounds(longitudes);
  const isGlobal = bounds.maximum - bounds.minimum > 300;
  return {
    isGlobal,
    longitudes: isGlobal ? normalizeLongitudes180(longitudes) : longitudes,
  };
}

export class IrregularVectorField implements TStreamlineVectorField {
  readonly isGlobal: boolean;
  readonly latitudeMin: number;
  readonly latitudeMax: number;
  readonly longitudeMin: number;
  readonly longitudeMax: number;
  readonly referenceSpeed: number;

  private readonly index: KDBush;
  private readonly regularField: RegularVectorField | undefined;
  private readonly latitudes: Float32Array;
  private readonly longitudes: Float32Array;
  private readonly uData: Float32Array;
  private readonly vData: Float32Array;
  private readonly interpolationRadiusKm: number;

  constructor(
    latitudes: Float32Array,
    longitudes: Float32Array,
    uData: Float32Array,
    vData: Float32Array
  ) {
    if (
      latitudes.length === 0 ||
      longitudes.length !== latitudes.length ||
      uData.length !== latitudes.length ||
      vData.length !== latitudes.length
    ) {
      throw new Error("Vector components do not match the unstructured grid");
    }
    const samples = filterFiniteVectorSamples(
      latitudes,
      longitudes,
      uData,
      vData
    );
    if (samples.latitudes.length === 0) {
      throw new Error("Vector field has no finite samples");
    }
    this.latitudes = samples.latitudes;
    const normalizedLongitudes = normalizeGlobalLongitudes(samples.longitudes);
    this.isGlobal = normalizedLongitudes.isGlobal;
    this.longitudes = normalizedLongitudes.longitudes;
    this.uData = samples.uData;
    this.vData = samples.vData;
    const latitudeBounds = finiteBounds(this.latitudes);
    const longitudeBounds = finiteBounds(this.longitudes);
    this.latitudeMin = Math.max(-89.5, latitudeBounds.minimum);
    this.latitudeMax = Math.min(89.5, latitudeBounds.maximum);
    this.longitudeMin = longitudeBounds.minimum;
    this.longitudeMax = longitudeBounds.maximum;
    this.interpolationRadiusKm = interpolationRadiusKm(
      this.latitudes.length,
      this.latitudeMin,
      this.latitudeMax,
      this.longitudeMin,
      this.longitudeMax
    );
    this.referenceSpeed = calculateReferenceSpeed(this.uData, this.vData);
    this.index = new KDBush(this.latitudes.length);
    for (let i = 0; i < this.latitudes.length; i++) {
      this.index.add(this.longitudes[i], this.latitudes[i]);
    }
    this.index.finish();
    this.regularField = this.createRegularField();
  }
....


// vectorField.test.ts

  it("samples global unstructured fields stored with 0..360 longitudes", () => {
    const latitudes: number[] = [];
    const longitudes: number[] = [];
    const uData: number[] = [];
    const vData: number[] = [];
    for (const latitude of [-1, 0, 1]) {
      for (let longitude = 0; longitude < 360; longitude++) {
        latitudes.push(latitude);
        longitudes.push(longitude);
        uData.push(longitude);
        vData.push(0);
      }
    }

    const field = new IrregularVectorField(
      Float32Array.from(latitudes),
      Float32Array.from(longitudes),
      Float32Array.from(uData),
      Float32Array.from(vData)
    );

    expect(field.isGlobal).toBe(true);
    expect(field.sample(0, -45)?.u).toBeCloseTo(315);
    expect(field.sample(0, -90)?.u).toBeCloseTo(270);
  });

@Karinon Karinon mentioned this pull request Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants