> ## Documentation Index
> Fetch the complete documentation index at: https://takeprofit.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# CSV format and limits

> Requirements for external CSV files in Indie: file structure, time formats, value parsing, hosting requirements, size limits and the error code reference.

This page lists the requirements for CSV files used as external data sources, how the files are fetched, and what
the error codes mean.

## File structure

* UTF-8 encoding; a leading BOM is tolerated.
* Comma (`,`) is the only supported delimiter — no semicolons, tabs or auto-detection.
* Standard CSV quoting (RFC 4180): a field containing commas, quotes or newlines must be enclosed in double quotes.
* Both LF and CRLF line endings work; blank lines are ignored.
* A header row is required. Column order does not matter, unknown extra columns are ignored.

## The time column

The time column (default name `time`, case-insensitive, configurable via `sources.Csv(url, time_column=...)`) accepts
exactly three formats:

| Format | Example |
| - | - |
| Unix timestamp, seconds | `1704153600` |
| Unix timestamp, milliseconds | `1704153600000` |
| RFC 3339 / ISO 8601 date-time | `2024-01-02T00:00:00Z`, `2024-01-02T09:30:00+02:00` |

Notes:

* Seconds vs milliseconds are distinguished by magnitude: values of 10<sup>12</sup> and above are read as
  milliseconds.
* A bare date without a time part (`2024-01-02`) is **not** accepted.
* Timezone offsets are normalized to UTC; timestamps without an explicit offset are not accepted.
* Timestamps must be **strictly increasing**: duplicates and out-of-order rows are rejected, nothing is sorted or
  deduplicated for you.
* The resolved date must fall between years 1900 and 2200.
* Candle CSVs do not accept sub-second timestamps. Typed-data CSVs have millisecond resolution.

## Cell values

| Field type | Accepted values |
| - | - |
| `float` | Decimal numbers, scientific notation (`1e5`). `NaN`, `Inf`, underscores (`1_000`) and hex are rejected. |
| `int` | Plain decimal integers within the 32-bit range. `1.0` is not a valid int. |
| `bool` | `true`, `false`, `1`, `0` (case-insensitive). |
| `str` | Any text, stored verbatim. |

**Missing values.** The only way to express "no value" is an empty (or whitespace-only) cell, and it is only allowed
in columns mapped to `Optional` fields. Literal markers like `NA`, `N/A`, `null` or `-` are not recognized — they are
parse errors in numeric and bool columns. In a candle CSV only the `volume` cell may be empty (it reads as 0).

**Validation is all-or-nothing.** A single bad cell — an unparseable timestamp, an empty required value, a malformed
number — rejects the entire file. Rows are never silently skipped or filled with defaults.

## Hosting requirements

* HTTPS only, with a valid certificate. Plain `http://` URLs are rejected.
* The file must be publicly reachable: no authentication headers can be sent, and URLs with embedded credentials
  (`https://user:pass@...`) are rejected. If you need access control, use signed URLs (the query string is part of
  the URL and is kept out of server logs).
* Up to 3 redirects are followed; every hop must also be HTTPS.
* Addresses on private and internal networks are blocked.
* gzip-compressed responses are supported; all size limits apply to the decompressed data.

## Fetching and caching

* The file is downloaded when the indicator instance is created, with a 30-second fetch timeout. The dataset is a
  frozen snapshot: it is not refreshed while the indicator runs.
* Downloads are cached on the server for about 10 minutes. Re-adding an indicator within that window may serve the
  previous content of the file.
* Requests to the same URL are throttled; under heavy concurrent use a fetch can fail with
  `external_rate_limited` — retry by re-adding the indicator.

## Limits

| Limit | Value |
| - | - |
| Maximum file size (after decompression) | 64 MiB |
| Maximum data rows | 1,000,000 |
| Maximum columns per row | 4,096 |
| Maximum size of one row | 1 MiB |
| Maximum parsed data size (typed CSVs) | 48 MiB |
| Maximum total size of string cells | 32 MiB |
| Maximum schema leaf fields | 64 |
| Fetch timeout | 30 s |

## Error reference

When an external source fails, the indicator shows the error code followed by a short explanation, for example
`external_invalid_number: high 5 is below low 7 (line 12)`. Errors found while reading the file name the CSV line
they come from. The codes are:

| Code | Meaning | What to check |
| - | - | - |
| `external_fetch_failed` | The file could not be downloaded: DNS failure, TLS error, timeout, non-success HTTP status, or a blocked address. | The URL is public, HTTPS, and responds within 30 s. |
| `external_rate_limited` | Too many fetches of external data in a short period. | Retry by re-adding the indicator. |
| `external_response_too_large` | The file exceeds the size limit. | Reduce the file below 64 MiB. |
| `external_too_many_rows` | More than 1,000,000 data rows. | Reduce the number of rows. |
| `external_too_many_columns` | A row exceeds 4,096 columns or 1 MiB. | Reduce the number of columns / row size. |
| `external_malformed_csv` | Broken CSV structure, or the URL returned an HTML page instead of a CSV file. | The URL serves the raw file, not a viewer or login page; quoting follows RFC 4180. |
| `external_empty` | No header row, no data rows, or (for candles) no rows in the requested chart range. | The file has a header and data covering the chart period. |
| `external_missing_column` | A required column is absent from the header. | Header names match: candle columns are case-insensitive, schema fields are case-sensitive, nested fields use dots. |
| `external_invalid_time` | A time cell could not be parsed, a candle timestamp has a sub-second part, or two increasing typed timestamps are closer than one millisecond. | Use Unix seconds/milliseconds or RFC 3339. |
| `external_time_not_increasing` | Duplicate or out-of-order timestamps. | Sort rows by time and remove duplicates. |
| `external_invalid_number` | A numeric cell could not be parsed, or candle prices are inconsistent (e.g. `high` below `low`, negative volume). | Numeric cells and OHLC relations. |
| `external_invalid_bool` | A bool cell is not `true`/`false`/`1`/`0`. | Bool cell values. |
| `external_int_out_of_range` | An int value does not fit into a 32-bit range. | Declare the field as `float`. |
| `external_missing_value` | An empty cell in a column mapped to a non-`Optional` field. | Fill the value or declare the field `Optional`. |
| `external_timeframe_mismatch` | Rows are more frequent than the `time_frame` declared at the call site. | The declared time frame matches the file cadence. |
| `external_schema_invalid`, `external_schema_too_large` | The declared dataclass schema is invalid or too large. | Schema rules in [Typed external data](https://takeprofit.com/docs/indie/External-data/Typed-external-data.md#declaring-a-schema). |
| `external_payload_too_large`, `external_string_data_too_large` | The parsed data exceeds 48 MiB, or string cells exceed 32 MiB in total. | Shorten or drop long string columns; reduce the number of rows. |
| `external_source_not_supported` | External data sources are currently disabled. | Try again later or contact support. |

## Errors when declaring the source

The URL itself is validated when the indicator instance is created, before any download, so these errors appear
without the file ever being fetched:

| Code | Meaning |
| - | - |
| `invalid_url` | The URL cannot be parsed as `https://host/...`, or contains a control character or an embedded space. |
| `scheme_not_https` | The URL scheme is not HTTPS. |
| `userinfo_not_allowed` | The URL embeds credentials (`https://user:pass@...`). |
| `ipv6_literal_not_supported` | The host is an IPv6 address literal. |
| `non_ascii_host` | The host contains characters outside ASCII. |
| `invalid_source_param` | The `time_column` argument is empty or contains control characters. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.