Input / Output (C++)

Sources

Source (parent)

class catana::io::Source

Parent Class Source to read data of type Point class.

Each child class must implement the two read functions. These must return the number of points which were read (i.e. put into the point array) and -1 if no more Points are available.

Subclassed by catana::io::GadgetSource, catana::io::PointContainerSource, catana::io::RawBinarySource< RecordType >, catana::io::TextSource< RecordType >

Public Functions

virtual size_t get_npoints()

Number of points which can be read. Not needed nessessarily.

virtual long long int read(PointContainer::iterator write_iterator, size_t n) = 0

Read next n points. Returns number of points put into write_iterator. -1 if EOF.

virtual long long int read(Point *write_iterator, size_t n) = 0

Read next n points. Returns number of points put into write_iterator. -1 if EOF.

virtual void reset() = 0

Reset the source, i.e. revert to initial state.

virtual ~Source()
PointContainer get_point_container()

Directly load all data into an PointContainer (no FilterStream needed).

PixelizedPointContainer get_pixelized_point_container(unsigned int nside)

Directly load all data into a PixelizedPointContainer with given NSide (no FilterStream needed).

TextSource

template <class RecordType>
class catana::io::TextSource

A Source class for text files. The Points are assumed to be stored in the RecordType Format.

Template Parameters
  • RecordType: A class defined in the record_types.hpp file. Determines the storage order in the binary file (cartesian, spherical, coordinate order)

Inherits from catana::io::Source

Public Functions

TextSource(std::string filename, double shift = 0)

Construct from filename.

Parameters
  • filename: filename of the text file
  • shift: shift coordinates along all axis (coordinates will be transformed x -> x+shift)

virtual long long int read(PointContainer::iterator write_iterator, size_t n)

Read next n points from file. Returns number of points put into point_s. -1 if EOF.

Parameters
  • write_iterator: iterator of PointContainer of points. [write_iterator, write_iterator + n] must be a valid range.
  • n: number of points to read

virtual long long int read(Point *write_iterator, size_t n)

Read next n points from. Returns number of points put into point_s. -1 if EOF.

Parameters
  • write_iterator: pointer to a Point[] C-array of points. [write_iterator, write_iterator + n] must be a valid range.
  • n: number of points to read

void reset()

Resets Source to initial state (everything can be read again)

GadgetSource

class catana::io::GadgetSource

A Source class for Binary Gadget files.

While reading, the coordinates will be automatically transformed from Mpc/h -> Mpc and the origin is moved to the center of the box.

Inherits from catana::io::Source

Public Functions

GadgetSource(std::string filename, bool verbose = false)

Construct Source from filename.

Parameters
  • filename: Data spread across multiple files will be automatically detected and loaded. Specify any of the files.
  • verbose: Set to true for additional output to stdout

long long int read(PointContainer::iterator write_iterator, size_t n)

Read next n points from file(s). Returns number of points put into point_s. -1 if EOF.

Parameters
  • write_iterator: iterator of PointContainer of Points. [write_iterator, write_iterator + n] must be a valid range.
  • n: number of points to read

long long int read(Point *write_iterator, size_t n)

Read next n points from file(s). Returns number of points put into point_s. -1 if EOF.

Parameters
  • write_iterator: pointer to a Point[] C-array of Points. [write_iterator, write_iterator + n] must be a valid range.
  • n: number of points to read

size_t get_npoints()

Total number of points which can be read.

void reset()

Resets Source to initial state (everything can be read again)

Public Static Attributes

const int skipsize

The size of the empty elements between blocks in the Gadget file (usually 4 bytes)

RawBinarySource

template <class RecordType>
class catana::io::RawBinarySource

A Source class for raw binary files. The Points are assumed to be stored in the RecordType Format.

Template Parameters
  • RecordType: A class defined in the record_types.hpp file. Determines the storage order in the binary file.

Inherits from catana::io::Source

Public Types

typedef RecordType record_t

Public Functions

RawBinarySource(std::string filename, bool verbose = true)

Construct from filename.

virtual long long int read(PointContainer::iterator write_iterator, size_t n)

Read next n points from file. Returns number of points put into point_s. -1 if EOF.

Parameters
  • write_iterator: iterator of PointContainer of points. [write_iterator, write_iterator + n] must be a valid range.
  • n: number of points to read

virtual long long int read(Point *write_iterator, size_t n)

Read next n points from. Returns number of points put into point_s. -1 if EOF.

Parameters
  • write_iterator: pointer to a Point[] C-array of points. [write_iterator, write_iterator + n] must be a valid range.
  • n: number of points to read

size_t get_npoints()

Total number of points which can be read (i.e. size of point_container)

void reset()

Resets Source to initial state (everything can be read again)

RawBinarySource(RawBinarySource const&)
RawBinarySource &operator=(RawBinarySource const&)

PointContainerSource

class catana::io::PointContainerSource

A wrapper around an PointContainer which lets it act as Source.

Inherits from catana::io::Source

Public Functions

PointContainerSource(const PointContainer &point_container)

Constructor from PointContainer.

long long int read(PointContainer::iterator write_iterator, size_t n)

Read next n points from point_container. Returns number of points put into point_s. -1 if EOF.

Parameters
  • write_iterator: iterator of PointContainer of Points. [write_iterator, write_iterator + n] must be a valid range.
  • n: number of points to read

long long int read(Point *write_iterator, size_t n)

Read next n points from point_container. Returns number of points put into point_s. -1 if EOF.

Parameters
  • write_iterator: pointer to a Point[] C-array of Points. [write_iterator, write_iterator + n] must be a valid range.
  • n: number of points to read

void reset()

Resets Source to initial state (everything can be read again)

size_t get_npoints()

Total number of points which can be read (i.e. size of point_container)

Sinks

Sink (parent)

class catana::io::Sink

Parent Class Sink to write data of type Point class.

Each child class must implement the two write functions. These must return the number of points which were written and -1 if the writing failed.

Subclassed by catana::io::PixelizedPointContainerSink, catana::io::PointContainerSink, catana::io::RawBinarySink< RecordType >, catana::io::TextSink< RecordType >

Public Functions

virtual long long int write(PointContainer::const_iterator read_iterator, size_t n) = 0

write points within [read_iterator, read_iterator+n) to sink.

Return
number of points written. -1 if failed
Parameters
  • read_iterator: [read_iterator, read_iterator+n) must be a valid range of points
  • n: number of points to read from read_iterator and write to sink

virtual long long int write(Point *read_iterator, size_t n) = 0

write points within [read_iterator, read_iterator+n) to sink.

Return
number of points written. -1 if failed
Parameters
  • read_iterator: [read_iterator, read_iterator+n) must be a valid range of points
  • n: number of points to read from read_iterator and write to sink

virtual ~Sink()
long long int write(const PointContainer &point_container)

write entire point_container directly to sink

Return
number of points written. -1 if failed
Parameters
  • point_container: source container to write

TextSink

template <class RecordType>
class catana::io::TextSink

A Sink class for text files. The Points will be stored in the RecordType Format.

Template Parameters
  • RecordType: A class defined in the record_types.hpp file. Determines the storage order in the binary file.

Inherits from catana::io::Sink

Public Types

typedef RecordType record_t

Public Functions

TextSink(std::string filename, bool verbose = true)

Construct from filename. File with “filename” will be created.

virtual long long int write(PointContainer::const_iterator read_iterator, size_t n)

write points within [read_iterator, read_iterator+n) to point_container.

Return
number of points written. -1 if failed
Parameters
  • read_iterator: [read_iterator, read_iterator+n) must be a valid range of points
  • n: number of points to read from read_iterator and write to sink

virtual long long int write(Point *read_iterator, size_t n)

write points within [read_iterator, read_iterator+n) to point_container.

Return
number of points written. -1 if failed
Parameters
  • read_iterator: [read_iterator, read_iterator+n) must be a valid range of points
  • n: number of points to read from read_iterator and write to sink

void close()

Close file (can no longer write)

TextSink(TextSink const&)
TextSink &operator=(TextSink const&)

RawBinarySink

template <class RecordType>
class catana::io::RawBinarySink

A Sink class for raw binary files. The Points will be stored in the RecordType Format.

Template Parameters
  • RecordType: A class defined in the record_types.hpp file. Determines the storage order in the binary file.

Inherits from catana::io::Sink

Public Types

typedef RecordType record_t

Public Functions

RawBinarySink(std::string filename, bool verbose = true, bool append = false)

Construct from filename. File with “filename” will be created.

~RawBinarySink()
virtual long long int write(PointContainer::const_iterator read_iterator, size_t n)

write points within [read_iterator, read_iterator+n) to point_container.

Return
number of points written. -1 if failed
Parameters
  • read_iterator: [read_iterator, read_iterator+n) must be a valid range of points
  • n: number of points to read from read_iterator and write to sink

virtual long long int write(Point *read_iterator, size_t n)

write points within [read_iterator, read_iterator+n) to point_container.

Return
number of points written. -1 if failed
Parameters
  • read_iterator: [read_iterator, read_iterator+n) must be a valid range of points
  • n: number of points to read from read_iterator and write to sink

void close()

Close file (can no longer write)

RawBinarySink(RawBinarySink const&)
RawBinarySink &operator=(RawBinarySink const&)

PointContainerSink

class catana::io::PointContainerSink

A wrapper around an PointContainer which lets it act as Sink.

Inherits from catana::io::Sink

Public Functions

PointContainerSink(PointContainer &point_container)

Construct from PointContainer. point_container will be filled with data.

long long int write(PointContainer::const_iterator read_iterator, size_t n)

write points within [read_iterator, read_iterator+n) to point_container.

Return
number of points written (=n)
Parameters
  • read_iterator: [read_iterator, read_iterator+n) must be a valid range of points
  • n: number of points to read from read_iterator and write to sink

long long int write(Point *read_iterator, size_t n)

write points within [read_iterator, read_iterator+n) to point_container.

Return
number of points written (=n)
Parameters
  • read_iterator: [read_iterator, read_iterator+n) must be a valid range of points
  • n: number of points to read from read_iterator and write to sink

Filters

Filter (parent)

class catana::io::Filter

Parent class Filter for filtering data of type Point.

Child classes need to implement bool filter(Point&), which returns true if point passes filtering and false if it needs to be removed.

Subclassed by catana::io::AngularMaskFilter, catana::io::GenericRadialWindowFunctionFilter, catana::io::TophatRadialWindowFunctionFilter

Public Functions

virtual bool filter(Point &point) = 0

Filtering function on Point. Returns true if point passes filter, false otherwise.

size_t operator()(PointContainer &point_container, bool resize = true)

Run filter on an point_container.

Return
number of points remaining after filtering (== point_container.size() if resize==true)
Parameters
  • point_container: point_container on which to apply filter
  • resize: if true, resize the point container. if false, the filtered data will be within [point_container.begin(), point_container.begin()+n), where n is the return value

virtual size_t operator()(Point *begin, Point *end)

Run filter on a C-array of Points.

Filter points between [begin, end). Remaining points will be stored between [begin, begin +n), where n is the return value

Return
number of points remaining after filtering.

TophatRadialWindowFunctionFilter

class catana::io::TophatRadialWindowFunctionFilter

A radial tophat window function filter, acceptance probability = 1 if r<R0, else 0.

Inherits from catana::io::Filter

Public Functions

TophatRadialWindowFunctionFilter(double R0)

Construct from scale parameter R0.

virtual bool filter(Point &point)

Filtering function on Point. Returns true if point passes filter, false otherwise.

GaussianRadialWindowFunctionFilter

class catana::io::GaussianRadialWindowFunctionFilter

A Gaussian window function filter: acceptance probability = exp(-(r/R0)^2)

Inherits from catana::io::GenericRadialWindowFunctionFilter

Public Functions

GaussianRadialWindowFunctionFilter(double R0)

Construct from scale parameter R0.

GenericRadialWindowFunctionFilter

class catana::io::GenericRadialWindowFunctionFilter

A generic radial window function filter, bases on a function returning the acceptance probability at a given radius.

Inherits from catana::io::Filter

Subclassed by catana::io::GaussianRadialWindowFunctionFilter

Public Functions

GenericRadialWindowFunctionFilter(std::function<double(double)> window_function)

Construct from window_function.

Parameters
  • window_function: a function with signature double(double r), where the return value is in [0,1] and gives the probability of an point at distance r passing the filter.

GenericRadialWindowFunctionFilter(std::function<double(double)> window_function, size_t interpolation_points, double min, double max, bool parallel_init = false, )

Construct from window_function, with interpolation (good if computation is intensive)

Parameters
  • window_function: a function with signature double(double r), where the return value is in [0,1] and gives the probability of an point at distance r passing the filter.
  • interpolation_points: number of points which are spaced equidistantly between [min, max] and between which the function will be linearly interpolated
  • min: lower bound of interpolation (runtime error if called with radius outside boundaries)
  • max: upper bound of interpolation (runtime error if called with radius outside boundaries)
  • parallel_init: if true, initialize interpolation points in parallel. Set to false if window_function is a Python point.

bool filter(Point &point)

Filtering function on Point. Returns true if point passes filter, false otherwise.

AngularMaskFilter

class catana::io::AngularMaskFilter

Applies an angular mask (float healpix map)

MASK==1: keep point at this pixel MASK==0: remove point at this pixel

Inherits from catana::io::Filter

Public Functions

AngularMaskFilter(std::string healpix_mask)

Construct from HEALPix map FITS file.

Parameters
  • healpix_mask: valid HEALPix FITS file, in RING format, with pixel values 1 (passes filter) and 0 (remove points)

AngularMaskFilter(const Eigen::ArrayXf &healpix_map)

Construct from HEALPix map array.

Parameters
  • healpix_map: valid HEALPix map, in RING format, with pixel values 1 (passes filter) and 0 (remove points)

bool filter(Point &point)

Filtering function on Point. Returns true if point passes filter, false otherwise.

FilterStreams

To combine a Source -> Filter -> Sink together, we can use a FilterStream

class catana::io::FilterStream

The FilterStream class combines source, filters and sink to a stream.

Public Functions

FilterStream(Source *source, Sink *sink, size_t buffer_size = 1000000, bool verbose = false)

Construct from Source and Sink.

Parameters
  • source: pointer to an instance of Source
  • sink: pointer to an instance of Sink
  • buffer_size: data will be read, filtered and stored in chuncks of buffer_size
  • subsample_size: size of the random subset taken from filtered data which is stored to sink. Set to 0 if no subsampling shall be done.
  • temp_file_name: If subsampling is turned on, name the filename of the temporary file to store data before subsampling
  • verbose: Set to true for additional output to stdout.

void set_source(Source *source)

Set the data source.

void add_filter(Filter *filter_p)

Add filter to the list of filters which will be applied during .run()

size_t run(size_t subsample_size = 0, std::string temp_filename = "tmp.bin")

Read data from source, filter it and write to sink.

Return
number of points written to sink.

size_t run_totemp(std::string temp_filename = "tmp.bin", bool append = true)

Run intermediate steps manually: write source to temporary file, apply filters.

This function is meant to use if multiple sources need to be combined and then subsampled (e.g. if data is spread across multiple files). Run the “run_totemp” function with the same temporary file for all sources first and then run the “run_fromtemp” function with the number of particles needed.

Return
number of points written to temporary file
Parameters
  • temp_filename: Path and name of the temporary file

size_t run_fromtemp(std::string temp_filename = "tmp.bin", size_t subsample_size = 0, bool remove_temp = true)

Run intermediate steps manually: write temporary file to sink (with subsampling)

Return
number of points written to sink
Parameters
  • subsample_size: The number of particles which are randomly drawn from the temporary file and moved to the sink point
  • temp_filename: Path and name of the temporary file

Examples