DataLogger Class Reference

DataLogger class for writing data to a Feather file using Apache Arrow. More...

#include <data_logger.h>

Public Member Functions
	DataLogger ()=default
	Default constructor.
virtual	~DataLogger ()
	Destructor.
void	add_metadata (const std::string &key, const std::string &value)
	Adds arbitrary string metadata to be embedded in the Feather file.
void	add_field (const std::string &name, std::shared_ptr< arrow::DataType > type)
	Adds a new field to the schema.
void	open_file (const std::string &filename)
	Opens the output file for writing.
void	set_value (const std::string &column_name, int64_t value)
	Sets the value for a specified column in the current row.
void	set_value (const std::string &column_name, int32_t value)
	Sets the value for a specified column in the current row.
void	set_value (const std::string &column_name, int16_t value)
	Sets the value for a specified column in the current row.
void	set_value (const std::string &column_name, int8_t value)
	Sets the value for a specified column in the current row.
void	set_value (const std::string &column_name, double value)
	Sets the value for a specified column in the current row.
void	set_value (const std::string &column_name, const std::string &value)
	Sets the value for a specified column in the current row.
void	set_value (const std::string &column_name, bool value)
	Sets the value for a specified column in the current row.
void	set_value_float16 (const std::string &column_name, float value)
	Sets the value for a specified column in the current row.
void	save_row ()
	Saves the current row to the Feather file.

Detailed Description

DataLogger class for writing data to a Feather file using Apache Arrow.

This class allows dynamic creation of a schema by adding fields prior to opening the file. Once the schema is defined, data rows can be built by setting individual column values and saved row-by-row into a Feather file using Zstd compression.

Constructor & Destructor Documentation

DataLogger()

DataLogger::DataLogger ( )

default

Default constructor.

~DataLogger()

DataLogger::~DataLogger ( )

virtual

Destructor.

Closes the Feather writer and the output file if they are open. Warnings are logged if either the writer or the file cannot be closed properly.

Member Function Documentation

add_field()

void DataLogger::add_field	(	const std::string &	name,
		std::shared_ptr< arrow::DataType >	type )

Adds a new field to the schema.

Adds a field with the specified name and data type to the internal schema. This must be called before the file is opened.

Parameters

name	The name of the field.
type	The Arrow data type for the field.

Exceptions

std::runtime_error

if called after the file has been opened.

add_metadata()

void DataLogger::add_metadata	(	const std::string &	key,
		const std::string &	value )

Adds arbitrary string metadata to be embedded in the Feather file.

This can be called any time before open_file().
If the same key is supplied more than once the value is overwritten.

Parameters

key	The metadata key (UTF-8, non-empty).
value	The metadata value (UTF-8, may be empty).

Exceptions

std::runtime_error

if the file has already been opened.

open_file()

void DataLogger::open_file

(

const std::string &

filename

)

Opens the output file for writing.

Constructs the schema from the added fields, attaches custom metadata (such as the program version), ensures that the parent directory exists, and opens the file for writing. A Feather writer is then created with Zstd compression. The current row values are initialized.

Parameters

filename

The path to the file to be written.

Exceptions

std::runtime_error

if no fields have been added or if file or writer initialization fails.

save_row()

void DataLogger::save_row

(

)

Saves the current row to the Feather file.

Constructs Arrow array builders for each column based on the field type, appends the current row's values (or nulls if missing), finalizes the arrays, and creates a record batch which is then written to the file. After writing, the row is reset to its default state.

Exceptions

std::runtime_error

if any step in appending data, finalizing arrays, or writing the record batch fails.

set_value() [1/7]

void DataLogger::set_value	(	const std::string &	column_name,
		bool	value )

Sets the value for a specified column in the current row.

Overloaded method for boolean values.

Parameters

column_name	The name of the column.
value	The boolean value to set.

Exceptions

std::runtime_error

if the file is not open or the column does not exist.

set_value() [2/7]

void DataLogger::set_value	(	const std::string &	column_name,
		const std::string &	value )

Sets the value for a specified column in the current row.

Overloaded method for string values.

Parameters

column_name	The name of the column.
value	The string value to set.

Exceptions

std::runtime_error

if the file is not open or the column does not exist.

set_value() [3/7]

void DataLogger::set_value	(	const std::string &	column_name,
		double	value )

Sets the value for a specified column in the current row.

Overloaded method for double values.

Parameters

column_name	The name of the column.
value	The double value to set.

Exceptions

std::runtime_error

if the file is not open or the column does not exist.

set_value() [4/7]

void DataLogger::set_value	(	const std::string &	column_name,
		int16_t	value )

Sets the value for a specified column in the current row.

Overloaded method for int16_t values.

Parameters

column_name	The name of the column.
value	The int16_t value to set.

Exceptions

std::runtime_error

if the file is not open or the column does not exist.

set_value() [5/7]

void DataLogger::set_value	(	const std::string &	column_name,
		int32_t	value )

Sets the value for a specified column in the current row.

Overloaded method for int32_t values.

Parameters

column_name	The name of the column.
value	The int32_t value to set.

Exceptions

std::runtime_error

if the file is not open or the column does not exist.

set_value() [6/7]

void DataLogger::set_value	(	const std::string &	column_name,
		int64_t	value )

Sets the value for a specified column in the current row.

Overloaded method for int64_t values.

Parameters

column_name	The name of the column.
value	The int64_t value to set.

Exceptions

std::runtime_error

if the file is not open or the column does not exist.

set_value() [7/7]

void DataLogger::set_value	(	const std::string &	column_name,
		int8_t	value )

Sets the value for a specified column in the current row.

Overloaded method for int8_t values.

Parameters

column_name	The name of the column.
value	The int8_t value to set.

Exceptions

std::runtime_error

if the file is not open or the column does not exist.

set_value_float16()

void DataLogger::set_value_float16	(	const std::string &	column_name,
		float	value )

Sets the value for a specified column in the current row.

Not overloaded method (like set_value methods), to avoid automatic promotions to double

Parameters

column_name	The name of the column.
value	The boolean value to set.

Exceptions

std::runtime_error

if the file is not open or the column does not exist.

---

The documentation for this class was generated from the following files:

• src/pogosim/data_logger.h
• src/pogosim/data_logger.cpp