Class HistoryExtractionState
Represents an object to be extracted from the source system. This object is intended to be used with a system that can both feed live data and read historical data.
This is intended to be used with more complicated setups involving multiple destination systems, and large amounts of historical data.
It works by keeping track of two TimeRange, SourceExtractedRange and DestinationExtractedRange.
SourceExtractedRange is the range of timestamps extracted from the source system, DestinationExtractedRange is a subset of SourceExtractedRange, representing the range of timestamps pushed to destination systems.
The intended mechanism of this state is to update the source range after each iteration of frontfill (reading forwards in time), and backfill (reading backwards), as well as handling "streamed" points, which may appear at any point in time.
Once these points are pushed to destinations, the DestinationExtractedRange is updated, this is the range that gets synchronized to state-stores.
On startup, the state can be initialized from one or more systems, and the most conservative number will be used. After calling InitExtractedRange from each destination system, FinalizeRangeInit must be called before reading history.
Implements
Inherited Members
Namespace: Cognite.Extractor.StateStorage
Assembly: Cognite.StateStorage.dll
Syntax
public class HistoryExtractionState : BaseExtractionState, IExtractionStateConstructors
| Edit this page View SourceHistoryExtractionState(string, bool, bool)
Create an extraction state with given id.
Declaration
public HistoryExtractionState(string id, bool frontfill = false, bool backfill = false)Parameters
| Type | Name | Description |
|---|---|---|
| string | id | Unique id of extraction state |
| bool | frontfill | True if this state will be frontfilled |
| bool | backfill | True if this state will be backfilled |
Properties
| Edit this page View SourceBackfillEnabled
True if this state has backfill enabled.
Declaration
public bool BackfillEnabled { get; protected set; }Property Value
| Type | Description |
|---|---|
| bool |
FrontfillEnabled
True if this state has frontfill enabled.
Declaration
public bool FrontfillEnabled { get; protected set; }Property Value
| Type | Description |
|---|---|
| bool |
IsBackfilling
True if state is actively backfilling, or should start.
Declaration
public bool IsBackfilling { get; protected set; }Property Value
| Type | Description |
|---|---|
| bool |
IsFrontfilling
True if state is actively frontfilling, or should start.
Declaration
public bool IsFrontfilling { get; protected set; }Property Value
| Type | Description |
|---|---|
| bool |
SourceExtractedRange
Range of data extracted from source system
Declaration
public TimeRange SourceExtractedRange { get; protected set; }Property Value
| Type | Description |
|---|---|
| TimeRange |
Methods
| Edit this page View SourceFinalizeRangeInit()
Called after range initialization to set uninitialized ranges to proper default values depending on whether backfill is enabled or not.
This means that uninitialized ranges (ranges that are complete after contracting), are set to (now, now) unless only frontfill is enabled, in which case it is set to (epoch, epoch).
Either way, SourceExtractedRange is set equal to DestinationExtractedRange, to start extracting from history.
If the destination range starts at epoch, backfilling is considered to be done.
Declaration
public virtual void FinalizeRangeInit()InitExtractedRange(DateTime, DateTime)
Called when initializing extracted range from destinations and state storage. This will always contract the believed range.
Declaration
public override void InitExtractedRange(DateTime first, DateTime last)Parameters
| Type | Name | Description |
|---|---|---|
| DateTime | first | |
| DateTime | last |
Overrides
| Edit this page View SourceRestartHistory()
Restart history by reseting the source range to the destination range, and resetting IsFrontfilling and IsBackfilling based on FrontfillEnabled and BackfillEnabled, as well as the destination range.
Declaration
public virtual void RestartHistory()UpdateDestinationRange(DateTime, DateTime)
Update the state with first and last points successfully pushed to destination(s). If backfill/frontfill is enabled, this will not extend the destination range outside of the first/last point in the source range. If backfill is enabled and done, and first is at or before the earliest extracted point, the destination range start will be set to zero.
Declaration
public override void UpdateDestinationRange(DateTime first, DateTime last)Parameters
| Type | Name | Description |
|---|---|---|
| DateTime | first | Earliest timestamp in successful push to destination(s) |
| DateTime | last | Latest timestamp in successful push to destination(s) |
Overrides
| Edit this page View SourceUpdateFromBackfill(DateTime, bool)
Update start of source range from history backfill. If final is true, IsBackfilling is set to false.
Declaration
public virtual void UpdateFromBackfill(DateTime first, bool final)Parameters
| Type | Name | Description |
|---|---|---|
| DateTime | first | Earliest timestamp in backfilled chunk |
| bool | final | True if this is the end of history |
UpdateFromFrontfill(DateTime, bool)
Update end of source range from history frontfill. If final is true, IsFrontfilling is set to false.
Declaration
public virtual void UpdateFromFrontfill(DateTime last, bool final)Parameters
| Type | Name | Description |
|---|---|---|
| DateTime | last | Last timestamp in frontfilled chunk |
| bool | final | True if this is the end of history |
UpdateFromStream(DateTime, DateTime)
Update source range from streamed points. Streamed data may come from anywhere. If backfill/frontfill is running, only points after/before the first/last timestamp are considered.
Declaration
public virtual void UpdateFromStream(DateTime first, DateTime last)Parameters
| Type | Name | Description |
|---|---|---|
| DateTime | first | First timestamp in streamed chunk |
| DateTime | last | Last timestamp in streamed chunk |