Class OperationScheduler<T>

Abstraction for handling some kind of recursive exploration. Option for shared, asynchronous limits, and using a shared TaskThrottler.

Inheritance

object

OperationScheduler<T>

SharedResourceScheduler<T>

Implements

IDisposable

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Namespace: Cognite.Extractor.Common

Assembly: Cognite.Common.dll

Syntax

public abstract class OperationScheduler<T> : IDisposable

Type Parameters

Name	Description
T	Type of item to explore

Constructors

OperationScheduler(IEnumerable<T>, TaskThrottler, int, CancellationToken)

Constructor

Declaration

public OperationScheduler(IEnumerable<T> initialItems, TaskThrottler throttler, int chunkSize, CancellationToken token)

Parameters

Type	Name	Description
IEnumerable<T>	initialItems	List of initial items, must be non-empty
TaskThrottler	throttler	TaskThrottler to use. Can be shared with other schedulers
int	chunkSize	Maximum number of items per chunk
CancellationToken	token	Cancellation token

Properties

TokenSource

Cancellation token source

Declaration

protected CancellationTokenSource TokenSource { get; set; }

Property Value

Type	Description
CancellationTokenSource

Methods

AbortChunk(IChunk<T>, CancellationToken)

Called if the scheduler is aborted before it finishes running. Handle any cleanup of resources related to the passed chunks. FreeCapacity is called outside of this method.

Declaration

protected abstract Task AbortChunk(IChunk<T> chunk, CancellationToken token)

Parameters

Type	Name	Description
IChunk<T>	chunk	Chunk to free
CancellationToken	token	Cancellation token

Returns

Type	Description
Task

ConsumeChunk(IChunk<T>, CancellationToken)

Method being called from TaskThrottler, operate on chunk and store the result so that it can be retrieved from the chunk later. Can safely throw exceptions, they are stored and can be handled in HandleTaskResult later.

Declaration

protected abstract Task ConsumeChunk(IChunk<T> chunk, CancellationToken token)

Parameters

Type	Name	Description
IChunk<T>	chunk	Chunk to consume
CancellationToken	token	Cancellation token

Returns

Type	Description
Task	Task

Dispose(bool)

Dispose method.

Declaration

protected virtual void Dispose(bool disposing)

Parameters

Type	Name	Description
bool	disposing

FreeCapacity(int)

Return used capacity allocated using GetCapacity.

Declaration

protected abstract void FreeCapacity(int freed)

Parameters

Type	Name	Description
int	freed	Amount of capacity to free

GetCapacity(int, bool)

Request to start reading requested nodes. Must return a number greater than 0 and less than requested. If shouldBlock is true it may block to wait for resources to be freed elsewhere.

Declaration

protected abstract Task<int> GetCapacity(int requested, bool shouldBlock)

Parameters

Type	Name	Description
int	requested	Maximum to request
bool	shouldBlock	True if this should block

Returns

Type	Description
Task<int>	Number greater than 0 and less than requested

GetChunk(IEnumerable<T>)

Construct a chunk object from a list of items.

Declaration

protected abstract IChunk<T> GetChunk(IEnumerable<T> items)

Parameters

Type	Name	Description
IEnumerable<T>	items	Items to construct chunk from

Returns

Type	Description
IChunk<T>

GetNextChunk(IEnumerable<T>, int, out IEnumerable<T>)

Get a single chunk from list. Should not return more items than capacity. Default implementation chunks by chunkSize and capacity.

Declaration

protected virtual IEnumerable<T> GetNextChunk(IEnumerable<T> items, int capacity, out IEnumerable<T> newItems)

Parameters

Type	Name	Description
IEnumerable<T>	items	Items to take from
int	capacity	Maximum number to return
IEnumerable<T>	newItems	New list after iterating. Can be a linq expression, does not need to maintain the same order, but should ensure that started items are placed first.

Returns

Type	Description
IEnumerable<T>	Items in a single new chunk

GetNextChunks(IEnumerable<T>, int, out IEnumerable<T>)

Get next chunks by consuming items from list. Should not get more items than capacity Default implementation simply calls GetNextChunk until it returns nothing or remaining capacity is 0.

Declaration

protected virtual IEnumerable<IChunk<T>> GetNextChunks(IEnumerable<T> items, int capacity, out IEnumerable<T> newItems)

Parameters

Type	Name	Description
IEnumerable<T>	items	Items to take from
int	capacity	Maximum number to return
IEnumerable<T>	newItems	New list after iterating. Can be a linq expression, does not need to maintain the same order, but should ensure that started items are placed first.

Returns

Type	Description
IEnumerable<IChunk<T>>	List of new chunks.

HandleTaskResult(IChunk<T>, CancellationToken)

Handle the result of a chunk operation. Called from the main loop after completion is reported. Returns a list of newly discovered items that should be scheduled.

Declaration

protected abstract Task<IEnumerable<T>> HandleTaskResult(IChunk<T> chunk, CancellationToken token)

Parameters

Type	Name	Description
IChunk<T>	chunk	Chunk to handle
CancellationToken	token	Cancellation token

Returns

Type	Description
Task<IEnumerable<T>>	New elements

OnIteration(int, int, int, int)

Called on each iteration of the scheduler loop, for reporting.

Declaration

protected abstract void OnIteration(int pending, int operations, int finished, int total)

Parameters

Type	Name	Description
int	pending	Number of items currently pending
int	operations	Number of operations that have been completed thus far
int	finished	Number of items that have been finished
int	total	Number of items that have been discovered in total

RunAsync()

Start the scheduler loop.

Declaration

public Task RunAsync()

Returns

Type	Description
Task	Task which terminates when the scheduler is finished

Implements

IDisposable

Extension Methods

DataModelUtils.GetOrCreateResourcesAsync<T, T2>(T2, IEnumerable<InstanceIdentifier>, Func<IEnumerable<InstanceIdentifier>, IEnumerable<SourcedNodeWrite<T>>>, Func<IEnumerable<SourcedNodeWrite<T>>, SanitationMode, (IEnumerable<SourcedNodeWrite<T>>, IEnumerable<CogniteError<SourcedNodeWrite<T>>>)>, int, int, RetryMode, SanitationMode, CancellationToken)