Namespace Cognite.Extractor.StateStorage

Classes

BaseExtractionState

A minimal extraction state implementation. Represents a single object in a source system that maps to a single object in some destination system.

Keeps track of a TimeRange DestinationExtractedRange which represents the range of timestamps that are currently present in the destination system.

BaseExtractionStatePoco

Represents an historical object in the destination system, used as storable object in state-stores.

BaseStorableState

Minimal state that may be stored to a state-storage system. Only contains an Id. This should be extended to create custom storable objects.

HistoryExtractionState

Represents an object to be extracted from the source system. This object is intended to be used with a system that can both feed live data and read historical data.

This is intended to be used with more complicated setups involving multiple destination systems, and large amounts of historical data.

It works by keeping track of two TimeRange, SourceExtractedRange and DestinationExtractedRange.

SourceExtractedRange is the range of timestamps extracted from the source system, DestinationExtractedRange is a subset of SourceExtractedRange, representing the range of timestamps pushed to destination systems.

The intended mechanism of this state is to update the source range after each iteration of frontfill (reading forwards in time), and backfill (reading backwards), as well as handling "streamed" points, which may appear at any point in time.

Once these points are pushed to destinations, the DestinationExtractedRange is updated, this is the range that gets synchronized to state-stores.

On startup, the state can be initialized from one or more systems, and the most conservative number will be used. After calling InitExtractedRange from each destination system, FinalizeRangeInit must be called before reading history.

LiteDBStateStore

Local database to store state between each run of the extractor. Used when the extractor needs to know the first/last points extracted from the source system and pushed to the destination.

StateStoreConfig

Configuration of state store to persist extraction state between runs

StateStoreProperty

Property to assign custom name in destination store. Default behavior transforms PascalCase into snake-case.

StateStoreUtils

Utilities used in state-storage

Interfaces

IExtractionState

Minimum required properties for an extraction state

IExtractionStateStore

Represents a general extractor state store, for storing first/last timestamps.

IRawDestination

A destination capable of pushing to raw

Enums

StateStoreConfig.StorageType

Enum for storage type in litedb