Loading Detector Data

The SmurfArchive module contains code for indexing and quickly loading detector data from an archive of streamed data written by the smurf-recorder OCS agent. A data archive should contain all timestream data recorded on a given system. The file paths within a data archive will look like:

<archive-path>/<date-code>/<stream-id>/<filename>.g3

For each stream-id, it is important that the time ranges of different files do not overlap or else data may not be returned in the correct order. This current version of the SmurfArchive system only works for a single stream-id, however future changes to the SmurfRecorder OCS agent and the SmurfArchive class will provide support for archives containing timestreams from multiple stream-id’s. This change will be necessary for systems with multiple smurf cards.

Indexing Timestream Data

To index data, the SmurfArchive module will walk through an archive and record information about each frame to an sqlite database, including the frame time, frame type, the number of channels and samples, etc. Only one index database is needed for each archive. The following snippet will write an index database to <archive_path>/frames.db or update an existing database with new frames if it already exists:

from so3g.smurf import SmurfArchive
arc = SmurfArchive('/data/timestreams')
arc.index_archive()

In this example, all detector data stored in /data/timestreams will be indexed and the index database will be written to /data/timestreams/frames.db.

A different database path can be specified by passing the db_path argument to the SmurfArchive.

Loading Data

To load detector data that includes the range [start, stop], use:

from so3g.smurf import SmurfArchive
arc = SmurfArchive('/data/timestreams')
d = arc.load_data(start, end)

The data will be loaded into a named_tuple which contains the detector data, primary data, and TES bias values, and metadata status. See the load_data Api for a more detailed description of allowed keyword arguments and the return value.

API

class so3g.smurf.SmurfArchive(archive_path, db_path=None, echo=False)[source]

Class to manage a smurf data archive.

Parameters

  • (path) (archive_path) – Path to the data directory

  • (path (db_path) – Path to the sqlite file. Defaults to <archive_path>/frames.db

  • optional) – Path to the sqlite file. Defaults to <archive_path>/frames.db

  • (bool (echo) – If true, all sql statements will print to stdout.

  • optional) – If true, all sql statements will print to stdout.

index_archive(verbose=False, stop_at_error=False)[source]

Adds all files from an archive to the sqlite database.

Parameters

  • verbose (bool) – Verbose mode

  • stop_at_error (bool) – If True, will stop if there is an error indexing a file.

load_data(start, end, show_pb=True, load_biases=True)[source]

Loads smurf G3 data for a given time range. For the specified time range this will return a chunk of data that includes that time range.

Parameters

  • (timestamp) (end) –

  • (timestamp) –

  • (bool (load_biases) –

  • optional) (If True, will return biases.) –

  • (bool –

  • optional) –

Returns

  • Returns a tuple SmurfData(times, data, primary, status, biases, timing_paradigm)

  • with the following fields –

    times (np.ndarray[samples]):

    Array of unix timestamps for loaded data

    data (np.ndarray[channels, samples]):

    Array of the squid phase in units of radians for each channel with data in the specified time range. The index of the array is the readout channel number.

    primary (Dict[np.ndarray]):

    Dict of numpy arrays holding the “primary” data included in the packet headers, including ‘AveragingResetBits’, ‘Counter0’, ‘Counter1’, ‘Counter2’, ‘FluxRampIncrement’, ‘FluxRampOffset’, ‘FrameCounter’, ‘TESRelaySetting’, ‘UnixTime’

    status (SmurfStatus):

    SmurfStatus object containing metadata info at the time of the first Scan frame in the requested interval. If there are no Scan frames in the interval, this will be None.

    biases (optional, np.ndarray[NTES, samples]):

    An array containing the TES bias values. If load_biases is False, this will be None.

    timing_paradigm(TimingParadigm):

    Tells you the method used to extract timestamps from the frame data.

load_status(time, show_pb=False)[source]

Returns the status dict at specified unix timestamp. Loads all status frames between session start frame and specified time.

Parameters

time (timestamp) – Time at which you want the rogue status

Returns

Dictionary of rogue variables at specified time.

Return type

status (dict)