An Enumerable CSV Processing Algorithm

The primary problem in converting a large CSV file into a list of objects is not finding the delimiter. The real cost comes from resolving column names again on every row, repeatedly searching for properties, loading the entire file into memory, and rewriting the same mapping code for different projects.

In this C# approach, which I developed entirely myself, the relationship between the data model and the file header is established only once. Subsequent rows are read from the stream and converted directly into objects through the previously prepared column-to-property mapping. The main reason the code remained useful across many projects despite its simplicity is this separation: variable work is moved into initialization, while repeated work follows the shortest practical execution path.

I was able to adapt this structure to different institutional data transfers very quickly. For a new file, the only requirement was usually to define a simple class representing its columns. The same core could process uncompressed text files and large GZip-compressed datasets with memory use independent of the total record count.

Schema mapping with reflection

The primary abstraction exposed by the algorithm is a generic data sequence:

\[ CSV<T> \]

T is the object type into which each row is converted. At startup, accessible properties of this type are obtained through reflection. PropertyInfo provides access to metadata such as a property's name, type, and assignment operation in .NET. Runtime assignment of values is performed through SetValue.

An important property of the code is that it does not restart reflection for every field. Type properties are examined once and stored in a dictionary under normalized names:

\[ \text{normalize(propertyName)} \rightarrow PropertyInfo \]

Normalization uses two simple rules:

Underscores are removed.

Letters are converted to lowercase in a culture-independent manner.

The following names can therefore reduce to the same mapping key:

CUSTOMER_ID CustomerId customer_id customerid

This approach tolerates small differences among naming conventions of different systems without requiring additional attributes or configuration files on the data model. In institutional transfer files, column names frequently originate from Java, C#, SQL, or legacy reporting systems. Ignoring underscore and letter-case differences substantially simplifies the mapping layer.

The objective is not to develop a general object-mapping framework. It is to connect known and controlled tabular data sources to a working processing pipeline quickly. In this scope, reflection is not abstraction overhead but a tool that shortens adaptation time.

Two-stage caching

Caching in the source code is not limited to retaining PropertyInfo objects in a dictionary. There are two actual precomputation stages.

In the first, property names are resolved:

\[ C_1: \text{property name} \rightarrow PropertyInfo \]

In the second, the CSV header is read and every column position is linked directly to the relevant property:

\[ C_2: \text{column index} \rightarrow PropertyInfo \]

Assume a file header is conceptually:

id;name;birth_date;source_code

If the data model contains only Id, Name, and BirthDate, the positional table becomes:

0 -> Id 1 -> Name 2 -> BirthDate 3 -> null

Column-name comparison is no longer performed while processing rows. Dictionary lookup is not repeated either. The loop accesses the PropertyInfo value directly at the column position. Unmapped columns are skipped because their entry is null.

This distinction reveals the actual value of caching. Not only reflection metadata, but also the combination of the file schema and object model is precomputed.

A naive implementation can search all properties again for every column in every row:

\[ O(NHP) \]

Where:

N is the number of rows.

H is the number of columns.

P is the number of properties.

Even with a dictionary, name-resolution cost remains in every cell. In the structure I developed, name resolution occurs only for the header:

\[ O(P+H) \]

During row processing, only positional array access is used:

\[ O(NH) \]

Reflection-based assignment remains in the hot path. Property discovery, name normalization, and column mapping, however, are removed entirely from data-row processing. This is the cache behavior that makes a difference on large data.

When Reset is called, the file is reopened and the header is read again, but the dictionary built from type properties is retained. Reflection discovery is therefore not repeated when the same enumerator is restarted.

Streaming data processing

The structure is built on the IEnumerable<T> and IEnumerator<T> contracts. CsvEnumerable<T> stores only the file path. Actual file opening and processing begin when an enumerator is requested. Each call to MoveNext reads one row and produces one T object.

IEnumerable<T> allows a dataset to be traversed sequentially through IEnumerator<T> and forms the fundamental contract of C# foreach.

The decisive consequence for large data is:

\[ \text{Memory use} \not\propto \text{total row count} \]

A file containing millions of rows is not loaded first into a List<T>. The calling code can process every object as soon as it is produced:

read the row create the object assign the fields process it move to the next row

This structure is particularly useful for:

Bulk database transfer

Statistical processing

Filtering

Data transformation

Analysis of large log and report files

Row-based export

Direct reading from GZip archives

If the caller stops traversal early, the rest of the file does not have to be read. When foreach completes or is interrupted, disposal of the enumerator closes the open reader and streams.

GZip and large-buffer use

If the filename ends in .gz, the source is opened through GZipStream. The compressed content is not fully extracted into a temporary file. The GZip layer and text reader are connected sequentially:

\[ \text{FileStream} \rightarrow \text{GZipStream} \rightarrow \text{StreamReader} \rightarrow \text{CSV row} \]

GZipStream is the .NET layer provided for streaming compression and decompression of gzip data. It permits sequential reading without retaining the full decompressed content in memory.

A one-mebibyte buffer is used for the file and text reader:

\[ 1,048,576 \text{ bytes} \]

This choice aims to reduce the number of operating-system reads made with small default buffers. Processing a large sequential file row by row is not the same as reading the physical source in small fragments. While the application advances by rows, the underlying reader can prefetch larger blocks.

The text reader defaults to UTF-8 without a BOM while enabling byte-order-mark detection. When an appropriate marker exists, this StreamReader constructor can recognize UTF-8, UTF-16, and UTF-32; otherwise, it uses the supplied encoding.

Opening the file with FileShare.ReadWrite makes it easier to read output that remains open in another process. This can be useful for live-generated files or files not yet closed by another system. It does not guarantee that a file changing during the read is transactionally consistent. The production protocol for such files must be defined separately.

Type-conversion behavior

A new T object is created for every row through a parameterless constructor. Activator.CreateInstance<T>() uses the parameterless constructor of the generic type at runtime.

The column value is then converted to the target property type with Convert.ChangeType. This method supports general conversion among basic .NET types. Conversion can be culture-sensitive. When no explicit format provider is supplied, the current runtime culture is used.

Conversion or assignment errors are caught at field level. One malformed value does not stop the entire row or file. A property that cannot be converted retains its default value, and processing continues with the remaining columns.

This behavior is useful in data collection and exploration. A single malformed date or numeric field does not discard a transfer containing millions of rows. A silent continuation policy is not sufficient by itself for financial or legal correctness, however. In such use, the number of failed fields, row positions, and conversion reasons should be recorded.

Although Convert.ChangeType provides a simple solution for basic types, additional converters may be required for:

Nullable types

Enum values

Guid

Culture-specific dates

Custom numeric formats

Application value objects

This requirement does not alter the core design. The conversion layer can be extended while preserving the column-property cache.

Complexity analysis

Let:

P be the number of model properties

H be the number of file columns

N be the number of data rows

L_i be the character length of a row

M be the number of mapped columns

Initialization cost is approximately:

\[ O(P+H+L_0) \]

Where L_0 is the header length.

For each data row, the algorithm performs:

Reading the row

Splitting by the delimiter

Creating one object

Converting and assigning mapped fields

Total runtime can be expressed as:

\[ O\left( P+H+\sum_{i=1}^{N}L_i+NM \right) \]

When column count is bounded, the algorithm behaves linearly in the total textual size of the file:

\[ \Theta(\text{total character count}) \]

Auxiliary memory cost is:

\[ O(P+H+L_{\max}) \]

Total row count does not appear in this expression. Only the current row, its split fields, the model object, and cache structures are retained.

Split allocates an array and field strings for every row. The algorithm is therefore not allocation-free, but it also does not accumulate memory in proportion to total file size. Short-lived objects can be reclaimed by the garbage collector. A newer implementation could reduce these allocations with Span<char>-based parsing. The main benefit of the existing code is rapid adaptability with low complexity.

Defining the CSV scope correctly

The code was developed for controlled delimited-text files. Tab, semicolon, comma, and vertical bar are tested in order in the header. The first character producing multiple fields is selected as the delimiter.

This automatic detection supports four common institutional export formats without configuration. It is not a general CSV-dialect detector. If a delimiter occurs inside a header value, the wrong character may be selected.

A more important boundary is that rows are split directly on the delimiter. In RFC 4180, fields can be enclosed in double quotes, and delimiters, quotes, and line breaks can occur as field content.

The core is therefore not a complete RFC 4180 parser handling:

A delimiter inside quotes

Escaped double quotes

A line break inside a field

A multiline text cell

I regarded this as a usage contract rather than a defect. In most of my projects, the data format was under my control or came from institutional exports whose fields did not contain delimiters or line breaks. In such a domain, the ReadLine and Split path was sufficient and fast without carrying the complexity of a full CSV state machine.

For a general-purpose library, the parsing layer can be replaced with a finite-state machine tracking quote state. The reflection cache, column mapping, GZip support, and streaming enumerator structure can remain unchanged.

Malformed-row policy

If the field count in a row does not match the header column count, the enumerator closes the file and terminates traversal. This behavior protects stream reliability rather than silently passing over a structural schema error.

There is a deliberate distinction between a field-level conversion error and row-structure corruption:

If one field cannot be converted, processing continues with the other fields.

If the row has an invalid column structure, the entire read terminates.

This prevents a shifted delimiter from assigning all subsequent values to incorrect properties. In a large transfer, controlled termination can be safer than producing thousands of incorrectly mapped objects.

The policy can be changed for different projects. A malformed row can be skipped, written to a quarantine file, or produced as an error object on a separate stream. The current core prefers a simple and conservative behavior.

Institutional value of simple code

The reason this approach remained useful across different projects was not only its speed. Its main value was that adapting it to new data required very few decisions.

The process for a new transfer was usually:

1. Examine the columns in the file

1. Define a simple corresponding data class

1. Adjust a few property names if necessary

1. Process the sequence with foreach

There is no need for custom mapper classes, per-row dictionaries, XML mapping files, or a large ORM layer. Unknown columns are ignored. Missing mappings leave the corresponding properties at default values. If file order changes, header mapping identifies the new positions again.

The cache mechanism combines performance optimization and developer productivity. Reflection information for the type is extracted once, the header is resolved once, and all remaining rows are processed positionally. Reflection flexibility is retained while the most expensive metadata searches are moved outside the loop.

A version requiring higher raw throughput could use precompiled setter delegates instead of PropertyInfo.SetValue. If type and column mapping remain unchanged throughout the application, the property cache can also be static per generic type. These are not alternatives to the current design, but more advanced applications of the same precomputation principle.

In my use, this small class provided a good balance between abstraction and actual requirements. It processed large files without loading them fully into memory, read compressed sources directly, adapted to changing column order, and could be moved into new projects quickly.

The main optimization is not a complex algorithm. It is precomputing everything that does not change from one row to the next. Reflection-based mapping becomes both flexible and sufficiently efficient for large data only when this distinction is made correctly.