Why your CSV loses your data types, and what to do about it

10 September 2026

A CSV keeps your values and nothing you knew about them. Which types, which units, which categories: none of it is in the file, so none of it comes back. This is about the ways round that, and about why the obvious ones stop working the moment a human has to open the file.

What a CSV cannot carry

A CSV is a grid of characters with a separator convention. It has no type system, so nothing in the file says what a column is. A reader has to infer each one from the text, and some things cannot be inferred at all because nothing in the file encodes them.

Take a table with a unit set on a column, and a column of grades that is a categorical:

T = table (datetime (2024, 1, 1) + hours ((0:1)'), ...
           categorical ({'high'; 'low'}), [1.5; 2.5], ...
           'VariableNames', {'when', 'grade', 'reading'});
T.Properties.VariableUnits = {'', '', 'kg'};

Write that to a plain CSV, read it back, and the two timestamps and the numbers return intact, because a reader can recognise those from their text. The rest does not:

  • grade comes back as a cell array of character vectors. Nothing in the file said it was categorical, and nothing recorded the set of categories, so the reader cannot know that high and low were two levels of one variable rather than two strings that happen to appear.
  • The unit is gone. So is any description you set, and so is the display format a datetime or duration was carrying.

None of that is a defect in anybody's CSV writer. There is nowhere in the file to put it. What you get back is the values, and your own memory of what they were.

One fix: write the types into the file

You can put them in the file yourself, as extra rows above the data. That is what this package's own CSV pair does:

"# varTypes 1 rows; varNames 1 rows; varDescriptions 0 rows; varUnits 1 rows.",
"datetime|dd-MMM-uuuu HH:mm:ss","double"
"when","reading"
"","kg"
"2024-01-01T00:00:00",1

Written by table2csv and read by csv2table, that round-trips exactly, units and display format included. It is a real answer and it costs nothing to use.

It also stops being a plain CSV. Hand that file to anything else, a spreadsheet application, pandas, R, and those three header rows arrive as data. You have solved the problem by making the file less portable, which is a trade rather than a fix.

The usual fix, and it is a good one

If you are moving data from one program to another, use a columnar binary format. Parquet is the common choice, Arrow underneath it. Types are part of the file rather than inferred from it, the reader is fast, and every language you are likely to use has one. For tool to tool work this is simply the right answer and I am not going to pretend otherwise.

It fails in exactly one situation, and it is a situation many of us are in more often than we admit.

When a person has to open the file

You cannot look at a Parquet file. You cannot mail it to a colleague who has LibreOffice and no Python. You cannot let a collaborator fix three typos in it and send it back. The moment a human being is in the loop, a binary format stops being an option, and most people fall back to CSV and accept the loss.

So the gap is not "lossless interchange", which is solved. The gap is lossless interchange in a file a person can open.

A spreadsheet, with the types on a hidden sheet

A spreadsheet already solves half of it. An OpenDocument file has typed cells, so a date is stored as a date and comes back as one, and the whole thing opens in LibreOffice by double-clicking it. What it has no place for is the rest of what you know about a table: which class each column is, what display format it carried, its units, its descriptions, which column holds the row labels.

The approach the datatypes package takes is to put that on a sheet of its own, named __datatypes_meta__, and leave the data sheets completely ordinary. A reader that knows nothing about the convention sees a normal spreadsheet. A reader that does gets the types back:

t  = datetime (2024, 1, 1) + hours ((0:2)');
TT = timetable (t, [1; 2; 3], 'VariableNames', {'reading'});
TT.Properties.VariableUnits = {'kg'};

timetable2ods (TT, 'measurements.ods');
TT2 = ods2timetable ('measurements.ods');

isequal (TT, TT2)      % true, units and row times included

Open measurements.ods in LibreOffice and you see three rows and two columns, the dates formatted as dates. Edit a number, save, read it back in Octave, and it is still a timetable with its units intact.

This is the part that the header-row approach cannot do. The metadata is in the file, but it is not in the way. The data sheet has exactly the rows your table has, so a colleague opening it sees a spreadsheet and not a spreadsheet with three rows of bookkeeping on top, and a tool that reads the sheet gets the data and nothing else. The hidden sheet is the only unusual thing in the file, and a spreadsheet application will not show it to you unless you go looking.

The same works for a plain table through table2ods and ods2table. For timetables carrying an attached event table, the event table is written as a further sheet and a cross-reference records the attachment, which makes this the only format in the package that carries events at all.

What this is not

It is not a standard, and I have deliberately not published it as one. It is a convention that one package writes, documented so that a file can be understood by somebody holding only the file, and so that a reader in another language could be written against it if anybody ever wants one. The layout is written up in ODS_FORMAT.md.

It is also not fast. A spreadsheet is XML in a zip container, and for a million rows you should be reaching for Parquet and not for this. The intended size is the size a person would open.

And there is an existing answer in this space worth knowing about: Frictionless Data's Table Schema puts a JSON descriptor beside a CSV, which is a standard, has implementations in several languages, and is the right choice if a sidecar file is acceptable to you. It is two files rather than one, and the two can be separated by any of the ordinary accidents of moving data around. That is the trade.

The short version

  • Program to program: Parquet. Types are in the file and it is fast.
  • A person has to open it, and you can accept two files: CSV plus a Table Schema descriptor.
  • A person has to open it, and it has to be one file: a spreadsheet with the types written into it, which is what the above does.
  • Careful with: a plain CSV you expect to carry more than values. Units, categories and formats are not in the file and will not come back.

The datatypes package for GNU Octave is developed by Andreas Bertsatos and funded by NLnet through the NGI Zero Commons Fund. Questions and corrections: the issue tracker.