src/app/core/common-components/entities-table/data-source/paginated-data-source.ts

Extends

EntitiesTableDataSource<T>

Relationships

Used by

No results matching.

Index

Properties
Methods
Accessors

Constructor

constructor()

Properties

Readonly hasUnknownTotalCount
Type : unknown
Default value : signal(false)

Whether the reported total is only a lower bound because there are more records than have been loaded/counted so far. In this case the paginator should indicate the total is unknown (e.g. "1 - 10 of 10+").

sortData
Type : unknown
Default value : () => {...}

The records arrive sorted from the database, so the table must not re-sort them (by different collation rules).

allRecords
Type : unknown
Default value : signal<T[]>([])
Inherited from EntitiesTableDataSource

All loaded records (only filled by data sources that hold all records in memory).

Readonly config
Type : unknown
Default value : signal<Readonly<Record<string, unknown>>>({})
Inherited from EntitiesTableDataSource

Options for this data source from the list or section config (the config of a dataSource: { name, config } entry), set by resolveDataSource. Data sources of feature modules read their own options from it; core data sources have none.

dataFilter
Type : unknown
Default value : signal<DataFilter<T>>({})
Inherited from EntitiesTableDataSource
Readonly effectiveSort
Type : unknown
Default value : signal<Sort | undefined>(undefined)
Inherited from EntitiesTableDataSource

The sort that the table applies (set by the table), as a signal: for data sources that arrange the rows themselves (the sort of the table only offers events).

Protected Readonly entityMapper
Type : unknown
Default value : inject(EntityMapperService)
Inherited from EntitiesTableDataSource
filteredRecords
Type : unknown
Default value : signal<T[]>([])
Inherited from EntitiesTableDataSource

The records passing all filters (including the free-text searchText), across all pages, in the order they were loaded (with server-side paging only the pages loaded so far, see getAllData). They are the input data of the table: sorting and pagination are applied on top of this by the table, see renderedRows.

isLoading
Type : unknown
Default value : signal(false)
Inherited from EntitiesTableDataSource
Protected loadGeneration
Type : number
Default value : 0
Inherited from EntitiesTableDataSource

Incremented whenever a new configuration triggers loading of new data. This allows stale requests to be detected.

loadRecordConfig
Type : unknown
Default value : signal<LoadRecordConfig<T>>(undefined)
Inherited from EntitiesTableDataSource
Readonly renderedRows
Type : unknown
Default value : toSignal(this.connect(), { requireSync: true })
Inherited from EntitiesTableDataSource

The records as currently rendered by the table: filtered, sorted and reduced to the current page.

Readonly searchText
Type : unknown
Default value : signal("")
Inherited from EntitiesTableDataSource

Free-text search for the records. Only data sources that support it apply it (see toolbarControls for the field to enter it); "Clear filters" resets it.

sortValueFns
Type : unknown
Default value : signal<SortValueFns<T>>({})
Inherited from EntitiesTableDataSource
Readonly toolbarControls
Type : Signal<DynamicComponentConfig[]>
Default value : signal([])
Inherited from EntitiesTableDataSource

controls a host may show in the list's filter row, in this order (default: none). Keep the config objects stable: a new object reloads the control.

Readonly usesPagination
Type : Signal<boolean>
Default value : signal(true)
Inherited from EntitiesTableDataSource

false: the table shows all rows and no paginator (may change, e.g. with an option of the data source)

Methods

Protected Async fetchPage
fetchPage(filter: DataFilter<T>, page: { limit: number; bookmark?: string }, sort: { prop?: string; dir?: "asc" | "desc" })

Fetch one page, continuing from page.bookmark when one is given.

The seam for data that is paged some other way than by a Mango query: the bookmark is opaque to this class, so an override is free to put its own cursor in it as long as a short page still means the end of the data.

Parameters :
Name Type Optional
filter DataFilter<T> No
page { limit: number; bookmark?: string } No
sort { prop?: string; dir?: "asc" | "desc" } No
Async getAllData
getAllData(filtered: unknown)
Inherited from EntitiesTableDataSource
Parameters :
Name Type Optional Default value
filtered unknown No false
Returns : Promise<T[]>
Protected Async loadRecords
loadRecords()
Inherited from EntitiesTableDataSource
Returns : unknown
Protected Async processEntityUpdate
processEntityUpdate()
Inherited from EntitiesTableDataSource
Returns : any
Protected resetPaginationCache
resetPaginationCache()

Discard all loaded records/cursor state and move back to the first page. Necessary whenever the query itself changes (filter, sort, page size) or the underlying data may have changed (entity update) - in all these cases the existing bookmark chain is no longer valid.

Returns : void
Protected listenToEntityUpdates
listenToEntityUpdates()
Inherited from EntitiesTableDataSource
Returns : void
rowOptionsOf
rowOptionsOf(_record: T)
Inherited from EntitiesTableDataSource

Optional extras for the row of a record (default: none). Called by the table for every rendered row: read signals here, so that the rows update when they change.

Parameters :
Name Type Optional
_record T No
Protected setRecords
setRecords()
Inherited from EntitiesTableDataSource

Request a (re)load of the records for the current config/filter/sort/page.

All the triggers that fire while a view initializes are debounced into a single request, so opening a list does not send several redundant DB requests (and the request already uses the fully resolved filter/sort/page). isLoading is set immediately so the table can show a progress indicator until the load finishes.

Returns : Promise<any>

a promise that resolves once the resulting load has completed.

Accessors

sort
getsort()
setsort(sort: MatSort)
Parameters :
Name Type Optional
sort MatSort No
Returns : void
paginator
getpaginator()
setpaginator(paginator: MatPaginator | null)
Parameters :
Name Type Optional
paginator MatPaginator | null No
Returns : void
import { Entity } from "#src/app/core/entity/model/entity";
import { MatSort } from "@angular/material/sort";
import { DataFilter } from "#src/app/core/filter/filters/filters";
import { MatPaginator } from "@angular/material/paginator";
import { effect, inject, signal } from "@angular/core";
import { EntityFilter } from "#src/app/core/filter/filters/entityFilter";
import { EntitiesTableDataSource } from "#src/app/core/common-components/entities-table/data-source/entities-table-data-source";
import { merge } from "rxjs";
import {
  EntitySpecialLoaderService,
  supportsPagination,
} from "#src/app/core/entity/entity-special-loader/entity-special-loader.service";
import {
  EntityPage,
  FULL_LOAD_PAGE_SIZE,
} from "#src/app/core/entity/entity-mapper/entity-mapper.service";
import { EntitySchemaService } from "#src/app/core/entity/schema/entity-schema.service";
import { EntitySchema } from "#src/app/core/entity/schema/entity-schema";
import { EntitySchemaField } from "#src/app/core/entity/schema/entity-schema-field";
import { getSchemaEmbedDatatype } from "#src/app/core/entity/schema/embedded-schema-field.util";
import { SchemaEmbedDatatype } from "#src/app/core/basic-datatypes/schema-embed/schema-embed.datatype";
import moment from "moment";

/** datatypes whose value is stored as the referenced id alone */
const STORED_BY_ID_DATATYPES = ["configurable-enum", "entity"];

export class PaginatedDataSource<
  T extends Entity,
> extends EntitiesTableDataSource<T> {
  private readonly specialLoader = inject(EntitySpecialLoaderService);
  private readonly schemaService = inject(EntitySchemaService);

  private sortRef: MatSort;
  private sortState: { prop?: string; dir?: "asc" | "desc" } = {};
  override set sort(sort: MatSort) {
    this.sortRef = sort;
    this.updateSort(this.sortRef.active, this.sortRef.direction);
    this.sortRef.sortChange.subscribe(({ active, direction }) => {
      this.updateSort(active, direction);
    });
  }
  override get sort(): MatSort {
    return this.sortRef;
  }

  /** The records arrive sorted from the database, so the table must not re-sort them (by different collation rules). */
  override sortData = (data: T[]) => data;

  private updateSort(active: string, direction: "asc" | "desc" | "") {
    if (
      (direction === "" && this.sortState.dir === undefined) ||
      (this.sortState.prop === active && this.sortState.dir === direction)
    ) {
      return;
    }
    if (direction === "") {
      this.sortState = {};
    } else {
      this.sortState = { prop: active, dir: direction };
    }
    this.resetPaginationCache();
  }

  private page: { size?: number; index?: number } = {};
  override set paginator(paginator: MatPaginator | null) {
    super.paginator = paginator;
    if (!paginator) {
      // the paginator was removed (see ListPaginatorComponent)
      return;
    }
    merge(paginator.initialized, paginator.page).subscribe(() => {
      this.page.size = paginator.pageSize;
      this.page.index = paginator.pageIndex;
      this.setRecords();
    });
  }
  override get paginator(): MatPaginator | null {
    return super.paginator;
  }

  /**
   * Whether the reported total is only a lower bound because there are more
   * records than have been loaded/counted so far. In this case the paginator
   * should indicate the total is unknown (e.g. "1 - 10 of 10+").
   */
  readonly hasUnknownTotalCount = signal(false);

  private effectiveFilter: DataFilter<T> = {};

  /** The DB cursor to continue fetching right after the records already in {@link filteredRecords}. */
  private bookmark: string | undefined;

  /** Whether the last fetch returned fewer records than requested, i.e. all matching records are already loaded. */
  private reachedEnd = false;

  constructor() {
    super();
    effect(() => {
      this.effectiveFilter = this.processFilterForDB(this.dataFilter());
      if (this.loadRecordConfig()) {
        this.resetPaginationCache();
      }
    });
  }

  /**
   * Discard all loaded records/cursor state and move back to the first page.
   * Necessary whenever the query itself changes (filter, sort, page size) or
   * the underlying data may have changed (entity update) - in all these
   * cases the existing bookmark chain is no longer valid.
   */
  protected resetPaginationCache() {
    this.filteredRecords.set([]);
    this.bookmark = undefined;
    this.reachedEnd = false;
    if (super.paginator) {
      // setting page index manually to not trigger the change event which would result in duplicate fetch
      super.paginator.pageIndex = 0;
      this.page.index = 0;
      this.setRecords();
    }
  }

  protected override async loadRecords() {
    if (!this.loadRecordConfig() || !super.paginator) {
      return [];
    }

    const start = this.page.size * this.page.index;
    // +1: a probe beyond the current page, to detect whether further pages exist
    const requiredLength = start + this.page.size + 1;
    const loadedLength = this.filteredRecords().length;

    if (loadedLength < requiredLength && !this.reachedEnd) {
      const deficit = requiredLength - loadedLength;
      const generation = this.loadGeneration;
      const res = await this.fetchPage(
        this.effectiveFilter,
        { limit: deficit, bookmark: this.bookmark },
        this.sortState,
      );
      if (generation !== this.loadGeneration) {
        // filter/sort/entity-update invalidated this request while it was in
        // flight - a new load for the current state is already under way.
        return [];
      }
      // update signal to trigger effect for data update in super-class
      this.filteredRecords.update((records) => [...records, ...res.records]);
      this.bookmark = res.bookmark;
      this.reachedEnd = res.records.length < deficit;
    }

    const totalLoaded = this.filteredRecords().length;
    this.hasUnknownTotalCount.set(totalLoaded > start + this.page.size);
    // `this.allRecords` stays empty;
  }

  /**
   * Fetch one page, continuing from `page.bookmark` when one is given.
   *
   * The seam for data that is paged some other way than by a Mango query: the
   * bookmark is opaque to this class, so an override is free to put its own
   * cursor in it as long as a short page still means the end of the data.
   */
  protected async fetchPage(
    filter: DataFilter<T>,
    page: { limit: number; bookmark?: string },
    sort: { prop?: string; dir?: "asc" | "desc" },
  ): Promise<EntityPage<T>> {
    const config = this.loadRecordConfig();
    if (supportsPagination(config.loaderMethod)) {
      // a loader that serves pages of its own: the pagination state stays here,
      // only the fetching moves
      return this.specialLoader.loadPageFor<T>(
        config.loaderMethod,
        config.forEntity,
        filter,
        page,
      );
    }
    return this.entityMapper.findType(config.entityCtr, filter, page, sort);
  }

  override async getAllData(filtered = false): Promise<T[]> {
    if (!this.loadRecordConfig()) return [];

    const entityCtr = this.loadRecordConfig().entityCtr;
    const filter = filtered ? this.effectiveFilter : {};

    // CouchDB's _find returns only 25 documents when no limit is given, so we
    // page through the results explicitly until a page is not completely filled.
    const allRecords: T[] = [];
    let exportBookmark: string | undefined;
    let page: T[];
    do {
      const res = await this.entityMapper.findType(
        entityCtr,
        filter,
        { limit: FULL_LOAD_PAGE_SIZE, bookmark: exportBookmark },
        this.sortState,
      );
      page = res.records;
      exportBookmark = res.bookmark;
      allRecords.push(...page);
    } while (page.length === FULL_LOAD_PAGE_SIZE);

    return allRecords;
  }

  protected override async processEntityUpdate() {
    // We don't really know how it might affect the pages -> full reload
    this.resetPaginationCache();
  }

  private processFilterForDB(filter: DataFilter<T>): EntityFilter<T> {
    // Mango queries need `$options: "i"` while CouchDB only supports `$regex: "(?i)..."`
    filter = convertToCouchRegex(filter);
    return this.toDbSelector(filter, {
      schema: this.loadRecordConfig()?.entityCtr?.schema,
    });
  }

  /**
   * Rewrite a query on the entity format into one on the database format.
   * The schema is walked alongside the query, so that fields nested inside
   * embedded fields (in `$elemMatch` or as dotted paths) are resolved as well.
   */
  private toDbSelector(query: unknown, ctx: SchemaContext): any {
    if (Array.isArray(query)) {
      return query.map((q) => this.toDbSelector(q, ctx));
    }
    if (!query || typeof query !== "object") {
      return query;
    }

    const result = {};
    for (const [key, value] of Object.entries(query)) {
      if (key.startsWith("$")) {
        result[key] = this.toDbSelector(value, ctx);
        continue;
      }
      const { dbPath, field } = this.toDbPath(key, ctx);
      const condition = this.toDbCondition(value, field);
      // pouchdb-find skips implicit equality with a falsy value (e.g. `{status: ""}`)
      // inside `$elemMatch` and matches every entry, an explicit `$eq` works on both engines
      result[dbPath] =
        ctx.inElemMatch && !isOperatorObject(condition)
          ? { $eq: condition }
          : condition;
    }
    return result;
  }

  /**
   * Translate a (dotted) field path into its path in the database format:
   * enum and entity references are stored as the plain id, so e.g.
   * "gender.id" becomes "gender" and "phoneNumber.type.id" becomes "phoneNumber.type".
   * Only for fields that are actually stored that way, though - on any other
   * field ".id" is a real path into an object value.
   */
  private toDbPath(
    key: string,
    ctx: SchemaContext,
  ): { dbPath: string; field?: EntitySchemaField } {
    const dbSegments: string[] = [];
    let current = ctx;
    let field: EntitySchemaField | undefined;
    for (const segment of key.split(".")) {
      if (
        segment === "id" &&
        STORED_BY_ID_DATATYPES.includes(field?.dataType)
      ) {
        continue;
      }
      dbSegments.push(
        current.container?.getEmbeddedFieldDbPath(segment) ?? segment,
      );
      field = current.schema?.get(segment);
      current = this.getEmbeddedContext(field);
    }
    return { dbPath: dbSegments.join("."), field };
  }

  private toDbCondition(
    condition: unknown,
    field: EntitySchemaField | undefined,
  ): unknown {
    if (condition === undefined) {
      // `undefined` cannot be sent in a Mango query, and dropping the key
      // would leave an empty selector that matches every document
      return { $exists: false };
    }
    if (!isOperatorObject(condition)) {
      return condition;
    }

    const result = {};
    for (const [operator, value] of Object.entries(condition)) {
      if (operator === "$elemMatch") {
        result[operator] = this.toDbSelector(value, {
          ...this.getEmbeddedContext(field),
          inElemMatch: true,
        });
      } else if (operator === "$not") {
        result[operator] = this.toDbCondition(value, field);
      } else {
        result[operator] = toDbDateBound(operator, value, field?.dataType);
      }
    }
    return result;
  }

  private getEmbeddedContext(
    field: EntitySchemaField | undefined,
  ): SchemaContext {
    const container = getSchemaEmbedDatatype(this.schemaService, field);
    return { schema: container?.getEffectiveSchema(field), container };
  }
}

/** The schema that the keys of a (nested) query object refer to. */
interface SchemaContext {
  schema?: EntitySchema;
  /** The datatype of the embedded field whose inner fields the keys refer to, which knows their paths in the stored value. */
  container?: SchemaEmbedDatatype;
  /** Whether the keys are the conditions on an array entry inside `$elemMatch`. */
  inElemMatch?: boolean;
}

function isOperatorObject(condition: unknown): condition is object {
  return (
    !!condition && typeof condition === "object" && !Array.isArray(condition)
  );
}

/**
 * Date filters compare with a day ("YYYY-MM-DD"), which records in memory match at day precision.
 * For a field stored in another format, rewrite the day into that format, keeping this meaning:
 * a "date" is a UTC timestamp, so the day becomes its first or last moment in local time;
 * a "month" is "YYYY-MM" and counts as its first day.
 */
function toDbDateBound(
  operator: string,
  value: unknown,
  dataType: string | undefined,
): unknown {
  if (
    (dataType !== "date" && dataType !== "month") ||
    !["$gte", "$gt", "$lte", "$lt"].includes(operator) ||
    typeof value !== "string"
  ) {
    return value;
  }
  const day = moment(value, "YYYY-MM-DD", true);
  if (!day.isValid()) {
    return value;
  }

  // `$lte` and `$gt` compare with the end of the day, `$gte` and `$lt` with its start
  const atEndOfDay = operator === "$lte" || operator === "$gt";
  if (dataType === "date") {
    if (atEndOfDay) {
      return day.endOf("day").toISOString();
    }
    // a date stored without time ("YYYY-MM-DD") counts as midnight UTC but sorts right before that timestamp,
    // so where the day starts at midnight UTC, only the day itself as bound treats it like in memory
    const startOfDay = day.startOf("day").toISOString();
    return startOfDay === `${value}T00:00:00.000Z` ? value : startOfDay;
  }
  return (atEndOfDay || day.date() === 1 ? day : day.add(1, "month")).format(
    "YYYY-MM",
  );
}

/**
 * Recursively converts any $regex value in an object to a CouchDB inline (?i) pattern.
 *
 * @param {*} node - The query object or value to transform.
 * @returns {*} The transformed query object.
 */
function convertToCouchRegex(node: any): any {
  if (node === null || typeof node !== "object") {
    return node;
  }

  if (Array.isArray(node)) {
    return node.map(convertToCouchRegex);
  }

  const result = {};

  for (const [key, value] of Object.entries(node)) {
    // Strip $options completely
    if (key === "$options") {
      continue;
    }

    // Convert $regex patterns to inline case-insensitive format
    if (key === "$regex" && typeof value === "string") {
      result["$regex"] = `(?i)${value}`;
    } else {
      result[key] = convertToCouchRegex(value);
    }
  }

  return result;
}

results matching ""

    No results matching ""