Class BaseExtractor<TConfig>

Base class for extractors.

Inheritance

object

BaseErrorReporter

BaseExtractor<TConfig>

Implements

IAsyncDisposable

Inherited Members

BaseErrorReporter.BeginWarning(string, string, DateTime?)

BaseErrorReporter.Warning(string, string, DateTime?)

BaseErrorReporter.BeginError(string, string, DateTime?)

BaseErrorReporter.Error(string, string, DateTime?)

BaseErrorReporter.BeginFatal(string, string, DateTime?)

BaseErrorReporter.Fatal(string, string, DateTime?)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Namespace: Cognite.Extractor.Utils.Unstable

Assembly: ExtractorUtils.dll

Syntax

public abstract class BaseExtractor<TConfig> : BaseErrorReporter, IAsyncDisposable

Type Parameters

Name	Description
TConfig	Config type.

Constructors

BaseExtractor(ConfigWrapper<TConfig>, IServiceProvider, ExtractorTaskScheduler, IIntegrationSink, CogniteDestination?)

Constructor, usable with dependency injection.

Declaration

public BaseExtractor(ConfigWrapper<TConfig> config, IServiceProvider provider, ExtractorTaskScheduler taskScheduler, IIntegrationSink sink, CogniteDestination? destination = null)

Parameters

Type	Name	Description
ConfigWrapper<TConfig>	config	Configuration object
IServiceProvider	provider	Service provider used to build this
ExtractorTaskScheduler	taskScheduler	Task scheduler.
IIntegrationSink	sink	Sink for extractor task updates and errors.
CogniteDestination	destination	Cognite destination.

Properties

Config

Configuration object

Declaration

protected TConfig Config { get; }

Property Value

Type	Description
TConfig

ConfigRevision

Currently active config revision.

Declaration

protected int? ConfigRevision { get; }

Property Value

Type	Description
int?

Destination

CDF destination

Declaration

protected CogniteDestination? Destination { get; }

Property Value

Type	Description
CogniteDestination

Provider

Access to the service provider this extractor was built from

Declaration

protected IServiceProvider Provider { get; }

Property Value

Type	Description
IServiceProvider

Scheduler

Scheduler for internal extractor tasks.

Use this for tasks that are not exposed to integrations, like state store, metrics, or other background processes.

Note that this is null until initialized in Init.

Declaration

protected PeriodicScheduler Scheduler { get; }

Property Value

Type	Description
PeriodicScheduler

Source

Cancellation token source.

Note that this is null until initialized in Init.

Declaration

protected CancellationTokenSource Source { get; }

Property Value

Type	Description
CancellationTokenSource

StartTime

Extractor start time. Set after Init has completed.

Declaration

protected DateTime? StartTime { get; }

Property Value

Type	Description
DateTime?

TaskScheduler

Task scheduler containing all public extractor tasks.

Declaration

protected ExtractorTaskScheduler TaskScheduler { get; }

Property Value

Type	Description
ExtractorTaskScheduler

Methods

AddMonitoredTask(Func<CancellationToken, Task<SchedulerTaskResult>>, string)

Add a task that should be watched by the extractor.

Use this for tasks that will not be reported to integrations, but that you still want to monitor, so that the extractor can crash if they fail or exit unexpectedly.

Declaration

protected void AddMonitoredTask(Func<CancellationToken, Task<SchedulerTaskResult>> task, string name)

Parameters

Type	Name	Description
Func<CancellationToken, Task<SchedulerTaskResult>>	task	Task to monitor.
string	name	Task name, just used for logging.

AddMonitoredTask(Func<CancellationToken, Task>, SchedulerTaskResult, string)

Add a monitored task that should be watched by the extractor.

Use this for tasks that will not be reported to integrations, but that you still want to monitor, so that the extractor can crash if they fail or exit unexpectedly.

This variant takes a static SchedulerTaskResult, to indicate whether the task is expected to terminate on its own or not.

Declaration

protected void AddMonitoredTask(Func<CancellationToken, Task> task, SchedulerTaskResult staticResult, string name)

Parameters

Type	Name	Description
Func<CancellationToken, Task>	task	Task to monitor.
SchedulerTaskResult	staticResult	Whether the task exiting on its own without cancellation should be considered an error.
string	name	Task name, just used for logging.

Exceptions

Type	Condition
ArgumentNullException

CancelMonitoredTaskAndWait(string)

Cancel a monitored task, then wait for it to complete.

This is typically used for ordered shutdown.

Declaration

protected Task CancelMonitoredTaskAndWait(string name)

Parameters

Type	Name	Description
string	name	Name of task to cancel.

Returns

Type	Description
Task

DisposeAsync()

Dispose the extractor asynchronously.

Declaration

public ValueTask DisposeAsync()

Returns

Type	Description
ValueTask

DisposeAsyncCore()

Dispose asynchronously, override this to clean up your resources on shutdown.

Prefer overriding ShutdownInternal instead or in addition to this method, if what you are doing is performing a graceful shutdown.

Typically, you will want to override this to call Dispose on any disposable resources, and ShutdownInternal to perform graceful cleanup.

Declaration

protected virtual ValueTask DisposeAsyncCore()

Returns

Type	Description
ValueTask

FlushSink(CancellationToken)

Flush the sink, writing any pending task events to integrations.

Declaration

protected Task FlushSink(CancellationToken token)

Parameters

Type	Name	Description
CancellationToken	token

Returns

Type	Description
Task

GetExtractorVersion()

Return the version of the active extractor.

Declaration

protected abstract ExtractorId GetExtractorVersion()

Returns

Type	Description
ExtractorId

Init(CancellationToken)

Initialize the extractor, if it has not already been initialized.

This is called automatically if you call Start, so only use this if you need to separate the init stage from the run stage, for example for testing.

Declaration

public Task Init(CancellationToken token)

Parameters

Type	Name	Description
CancellationToken	token	Cancellation token to use for the run

Returns

Type	Description
Task

InitTasks()

Initialize the extractor, adding tasks to the task runner as needed.

This runs before the extractor reports startup, if you have complex or heavy startup tasks, they should run in one or more tasks in the task scheduler, set to run immediately on startup.

The task runner is not started yet when this method is called.

Declaration

protected abstract Task InitTasks()

Returns

Type	Description
Task

NewError(ErrorLevel, string, string?, DateTime?)

Create a new extractor error belonging to this reporter.

Declaration

public override ExtractorError NewError(ErrorLevel level, string description, string? details = null, DateTime? now = null)

Parameters

Type	Name	Description
ErrorLevel	level	Error level.
string	description	Short error description.
string	details	Long error details.
DateTime?	now	Optional current timestamp.

Returns

Type	Description
ExtractorError

Overrides

BaseErrorReporter.NewError(ErrorLevel, string, string?, DateTime?)

Shutdown()

Shut down the extractor.

This calls ShutdownInternal then cancels the token.

If you wish to change shutdown behavior, override ShutdownInternal.

Declaration

public Task Shutdown()

Returns

Type	Description
Task

ShutdownInternal()

Perform graceful shutdown.

By default this method cancels the task scheduler and flushes the sink, you may wish to override this entirely to perform a different sequence of cleanup tasks.

Shutdown is required to be idempotent, and should not throw exceptions.

Declaration

protected virtual Task ShutdownInternal()

Returns

Type	Description
Task

Start(CancellationToken)

Start the extractor and wait for it to finish.

Declaration

public Task Start(CancellationToken token)

Parameters

Type	Name	Description
CancellationToken	token

Returns

Type	Description
Task

TestConfig()

Verify that the extractor is configured correctly.

Does nothing by default.

Declaration

protected virtual Task TestConfig()

Returns

Type	Description
Task	Task

Implements

IAsyncDisposable

Extension Methods

DataModelUtils.GetOrCreateResourcesAsync<T, T2>(T2, IEnumerable<InstanceIdentifier>, Func<IEnumerable<InstanceIdentifier>, IEnumerable<SourcedNodeWrite<T>>>, Func<IEnumerable<SourcedNodeWrite<T>>, SanitationMode, (IEnumerable<SourcedNodeWrite<T>>, IEnumerable<CogniteError<SourcedNodeWrite<T>>>)>, int, int, RetryMode, SanitationMode, CancellationToken)