Class Chunking
Support for splitting sequences into sequences of sequences subject to various constraints, as well as running a sequence of tasks in parallel.
Inherited Members
Namespace: Cognite.Extractor.Common
Assembly: Cognite.Common.dll
Syntax
public static class ChunkingMethods
| Edit this page View SourceChunkByHierarchy<T, K>(IEnumerable<T>, int, Func<T, K>, Func<T, K>, IEqualityComparer<K>?)
Chunk a list of entries with some sort of parent-child hierarchy into an ordered list where each element
contains all the parents of the next. The first list has no parents, or the parents are not in the list of given elements.
If maxSize is set larger than 1, the lists are merged so that the first list may contain both
the first and second layers, etc. respecting the maximum number of elements in each chunk.
The final lists may be larger than maxSize, it is only used to merge layers.
This is useful when pushing large numbers of assets to CDF.
Declaration
public static IEnumerable<IEnumerable<T>> ChunkByHierarchy<T, K>(this IEnumerable<T> input, int maxSize, Func<T, K> idSelector, Func<T, K> parentIdSelector, IEqualityComparer<K>? comparer = null) where K : notnullParameters
| Type | Name | Description |
|---|---|---|
| IEnumerable<T> | input | Input list |
| int | maxSize | Maximum number of elements when merging lists |
| Func<T, K> | idSelector | Function to get id from each element |
| Func<T, K> | parentIdSelector | Function to get parentId from each element |
| IEqualityComparer<K> | comparer | Optional equalitycomparer |
Returns
| Type | Description |
|---|---|
| IEnumerable<IEnumerable<T>> | A list of lists so that if they are consumed in order, all parents will be pushed ahead of children |
Type Parameters
| Name | Description |
|---|---|
| T | Element to be grouped |
| K | Type used for ids |
ChunkBy<T>(IEnumerable<T>, int)
Chunk the input sequence into a sequence of sequences subject to a constraint on the maximum number of elements per inner sequence.
Declaration
public static IEnumerable<IEnumerable<T>> ChunkBy<T>(this IEnumerable<T> input, int maxSize)Parameters
| Type | Name | Description |
|---|---|---|
| IEnumerable<T> | input | List of objects to be chunked |
| int | maxSize | Maximum number of entires per list in the output |
Returns
| Type | Description |
|---|---|
| IEnumerable<IEnumerable<T>> | A list of lists where each list is subject to |
Type Parameters
| Name | Description |
|---|---|
| T | Input type |
ChunkBy<TKey, TVal>(IEnumerable<(TKey, IEnumerable<TVal>)>, int, int)
Chunk the input [(key, [values])] sequence into a sequence subject to constraints on the number of keys and total number of values per chunk. Keys should be unique.
Declaration
public static IEnumerable<IEnumerable<(TKey Key, IEnumerable<TVal> Values)>> ChunkBy<TKey, TVal>(this IEnumerable<(TKey, IEnumerable<TVal>)> points, int maxPerList, int maxKeys)Parameters
| Type | Name | Description |
|---|---|---|
| IEnumerable<(TKey, IEnumerable<TVal>)> | points | List of (key, [values]) that will be chunked |
| int | maxPerList | Maximum number of entries per list of values in the output |
| int | maxKeys | Maximum number of keys per list of groups in the output |
Returns
| Type | Description |
|---|---|
| IEnumerable<IEnumerable<(TKey Key, IEnumerable<TVal> Values)>> | A chunking of the output subject to |
Type Parameters
| Name | Description |
|---|---|
| TKey | Type of keys to group by, should be unique |
| TVal | Value type |
RunThrottled(IEnumerable<Func<Task>>, int, Action<Task>?, CancellationToken)
Runs the generated tasks in parallell subject to the maximum parallelism.
Call the taskCompletedCallback action for each completed task
Declaration
public static Task RunThrottled(this IEnumerable<Func<Task>> generators, int parallelism, Action<Task>? taskCompletedCallback, CancellationToken token)Parameters
| Type | Name | Description |
|---|---|---|
| IEnumerable<Func<Task>> | generators | Tasks to perform |
| int | parallelism | Number of tasks to run in parallel |
| Action<Task> | taskCompletedCallback | Action to call on task completion |
| CancellationToken | token | Cancellation token |
Returns
| Type | Description |
|---|---|
| Task |
RunThrottled(IEnumerable<Func<Task>>, int, CancellationToken)
Runs the generated tasks in parallell subject to the maximum parallelism.
Declaration
public static Task RunThrottled(this IEnumerable<Func<Task>> generators, int parallelism, CancellationToken token)Parameters
| Type | Name | Description |
|---|---|---|
| IEnumerable<Func<Task>> | generators | Tasks to perform |
| int | parallelism | Number of tasks to run in parallel |
| CancellationToken | token | Cancellation token |
Returns
| Type | Description |
|---|---|
| Task |