frame_input
Mojo module đźˇ
frame_input
Reading a dataframe_mojo column into the List[Float64]/ List[String] the rest of this package draws from (#364).
A dataframe column is a payload buffer plus an Arrow-style validity
bitmap, so two things have to come out of it: the values, and which of
them are there at all. Both come out, and what happens to a gap follows
Theme.missing (#367):
- A value column’s invalid slots become
NaN, the marker every column carries inside aPlot. The marks draw around it – a line breaks, a point is not drawn. - A category column’s invalid slots become
Theme.missing_category_label, because “not recorded” is a label a reader can use, and dropping those rows would throw away observations whose value is perfectly good. - Under
Missing.RAISEboth are refused at the boundary, naming the column and the first row.
Values are read straight out of Arrow’s buffers – Column. unsafe_values() for the payload, is_valid for the bit – rather than
through Series.get, which boxes every element into an AnyValue
carrying a String field a numeric column never fills and rescans the
dtype list per call. Upstream measured that difference at 1,000,000
rows: about 3,500 us through get against about 700 us reading the
buffers (dataframe_mojo#91, #99). Column.value(i) is slower still and
is not used here. A string column reads the same way, through
Series.string() and StringColumn.to_list(), which walks the
large_utf8 bytes and offsets.
One copy remains, into the List[Float64] a Plot owns, because a
plot outlives the frame it was built from. The Series is kept alive
in each function for as long as its pointer is read, which is what the
buffer accessors require.