Skip to content
frame_input

frame_input

Mojo module 🡭

frame_input

Reading a dataframe_mojo column into the List[Float64]/ List[String] the rest of this package draws from (#364).

A dataframe column is a payload buffer plus an Arrow-style validity bitmap, so two things have to come out of it: the values, and which of them are there at all. Both come out, and what happens to a gap follows Theme.missing (#367):

  • A value column’s invalid slots become NaN, the marker every column carries inside a Plot. The marks draw around it – a line breaks, a point is not drawn.
  • A category column’s invalid slots become Theme.missing_category_label, because “not recorded” is a label a reader can use, and dropping those rows would throw away observations whose value is perfectly good.
  • Under Missing.RAISE both are refused at the boundary, naming the column and the first row.

Values are read straight out of Arrow’s buffers – Column. unsafe_values() for the payload, is_valid for the bit – rather than through Series.get, which boxes every element into an AnyValue carrying a String field a numeric column never fills and rescans the dtype list per call. Upstream measured that difference at 1,000,000 rows: about 3,500 us through get against about 700 us reading the buffers (dataframe_mojo#91, #99). Column.value(i) is slower still and is not used here. A string column reads the same way, through Series.string() and StringColumn.to_list(), which walks the large_utf8 bytes and offsets.

One copy remains, into the List[Float64] a Plot owns, because a plot outlives the frame it was built from. The Series is kept alive in each function for as long as its pointer is read, which is what the buffer accessors require.