Turning SEC filings into markdown in seconds

EDGAR does not give you a 10-K. It gives you a submission: one accession, many exhibits, mixed HTML, PDF, XML, graphics, and an XBRL package, wrapped in 1990s SGML. If you GET the “primary document” HTML and run a generic readability extractor, you lose exhibits, you smash tables, and you are already minutes behind acceptance.

This is how we turn that package into markdown chunks that Document Search can query, and that Document Contents can stitch back into a readable body — usually while the filing is still on the current-filings feed.

There is no webhookCopy Anchor Link

The SEC does not push new accessions. The current-filings Atom feed is the closest thing to a firehose:

https://www.sec.gov/cgi-bin/browse-edgar?action=getcurrent&...&count=10000&output=atom

We poll it on a ~5 second cycle (plus a few hundred milliseconds of jitter so a fleet of pollers does not synchronize). Each entry carries an index URL, a title, and an updated stamp. The accession is parsed out of the URL (NNNNNNNNNN-YY-NNNNNN). Dedup is (accession, updated) — the same accession can come back when EDGAR revises the index, and we want the correction, not a skip.

That is the entire detection story. Acceptance datetime in the SGML header is metadata we parse later. It is not how we learn that a filing exists. If you wait on submissions.zip or a daily full-index scrape, you are in the historical path, not the live one.

Fair access is a shared token bucket across the poller and every download worker (we run on the order of 8 requests/second, burst 8). The feed GET, the index GET, and the .txt GET all compete for the same budget. Under a busy 8-K hour, queue depth — not parse time — is what you feel.

We only run the live poller in the Eastern filing window (roughly 06:00–22:00 ET on business days, with a short pre-window warmup). Outside that, the same workers drain missing accessions against the submissions inventory so the corpus does not have holes.

Fetch the whole submission, not the pretty HTMLCopy Anchor Link

A worker that dequeues a live accession does two GETs:

1. The filing index HTML.
2. The complete submission {accession}.txt — the SGML file listed in the “Document Format Files” table.

The .txt is the source of truth. It contains the header (ACCESSION-NUMBER, ACCEPTANCE-DATETIME, filer CIKs, FILING-DATE, …) and every <DOCUMENT>…<TEXT>…</TEXT></DOCUMENT> exhibit, often UU-encoded. If you only download aapl-20240928.htm you cannot reconstruct siblings, you miss XML forms, and you cannot tell HTML and PDF that are the same exhibit.

We locate the .txt href with a table parse first (the summary is Document Format Files), regex as fallback. Then we parse SGML in memory: metadata until the first <DOCUMENT>, then each document block. Binary-ish payloads are UU-decoded when the headers say so.

ACCEPTANCE-DATETIME is YYYYMMDDHHMMSS in US Eastern wall clock. EDGAR sometimes serializes it with a fake Z. We treat the digits as Eastern, including DST, and store a real UTC instant. Date-only filing dates are midnight Eastern. That UTC instant is filed_at on every Document item. Search from_date / to_date are calendar dates on the Eastern acceptance day, not a UTC-day slice of filed_at.

Route exhibits; do not dump the zipCopy Anchor Link

Each SGML document has TYPE, SEQUENCE, FILENAME, DESCRIPTION, and a body. We do not markdown-chunk everything.

HTML wins over PDF for the same (DESCRIPTION, TYPE). EDGAR routinely ships both. If we chunked both, search would duplicate every paragraph and document_id would fork. The PDF is recorded as a sibling file; it reuses the HTML’s chunk stream.

SEQUENCE is the exhibit order in the submission (the 10-K is usually the lowest sequence that is not an attachment). The integer we use in document_id is not SEQUENCE. It is a counter of documents that actually produced chunks in this parse — so skipped graphics do not punch holes in the id space. Public ids look like:

• Filing: the accession (0000320193-24-000123)
• Exhibit: accession|chunked_document_id (and accession|chunk when we have to mint an exhibit key)
• Chunk (internal): accession|chunked_document_id|chunk_idx

Document Reference with a bare accession returns kind=filing: document_id is the primary file, and documents[] is every exhibit. Pass a document_id from Search or from documents[] when you want one file (kind=document, slim parent, no sibling list). filing_id stays the parent accession. That split — filing vs exhibit vs chunk — is the usual reason an Archives URL 404s: the folder uses the integer CIK (320193), not the zero-padded 0000320193 you store on the company.

HTML to markdown is a filing problem, not a blog problemCopy Anchor Link

A 10-K is not an article. It is nested tables, page markers, EDGAR <FONT> soup, and headers that are visually h2 but semantically a <b> in a <p>.

We parse with an XML-tolerant HTML tree, strip script / style / img, unwrap empty links, drop display:none nodes (they are often iXBRL scaffolding or hidden headers). Then a custom HTML→markdown converter — not a readability grab of the “main” div. Tables become pipe tables. Headings become ATX (# … ######). We then clean: drop empty columns and rows, fix broken pipes, collapse runaway blank lines. Junk in, junk out: if the HTML table has no header row, the markdown will not invent one.

PDF is a worse fallback: page text extraction in reading order, then the same chunker. It is not a PDF-to-markdown model. Prefer HTML.

XML forms are not markdown-native. We run the document through the SEC stylesheet that matches form type + schema version (with a fallback ladder when the version on the document disagrees with the filing). Small XML becomes HTML and then the same markdown path. Large XML (on the order of 0.7 MB+) skips the pretty converter and goes through a fast text/table chunker — N-PORT-scale files will otherwise dominate the live queue.

Chunking is header-aware on purposeCopy Anchor Link

A 200-page 10-K as one string is useless for search and too big for a context window. Naive split-every-N-characters cuts a table in half and loses the section title.

Defaults we actually use:

Algorithm, in order:

1. Split the markdown into header groups. Consecutive ATX lines (and blank lines that only separate headers) are the title of the following body. A 10-K “Item 1A. Risk Factors” heading stays attached to the risks, not to the previous item.
2. Split each body into blocks: markdown tables (lines with |), paragraphs, blank separators.
3. Pack blocks toward 2048. A small leftover block may ride along up to the cap if it is under the overlap budget.
4. Oversized tables are split by row, but every partition repeats the header row (and the --- separator when present). Otherwise “Q3 revenue” in row 400 is a pipe row with no column names.
5. Oversized paragraphs break on ., then ,, then space, then a hard cut. We keep original whitespace; this is not a tokenizer.

Only the first chunk of a header segment keeps the heading text. Later chunks of the same Item 8 do not repeat “Item 8” unless a new header group starts. That is a deliberate trade: less duplication in the index, slightly more work for a reader of chunk 4. Document Contents concatenates in chunk_idx order and merges adjacent markdown, so the stitched exhibit reads as one document again.

Each chunk we store carries chunk_idx, total_chunks, the exhibit’s sequence, filenames, acceptance time, and the parent accession. Embeddings are optional and off by default on the hot path — ingest latency matters more than a vector on second 2.

Corrections and backfillCopy Anchor Link

If the SGML header says CORRECTION (or a timestamp restatement), we delete the previous chunk set for that accession and write the new parse. Partial updates of a single exhibit are how you get two “Item 1”s in search.

Historical coverage is a different producer: walk submissions.zip (or a local copy), diff against accessions we already have, enqueue with a lower priority than live. You can shard that walk across workers. You can skip re-chunking when the exhibit types on disk already match what we stored. The live poller and the gap-filler share the same parse/transform/flush so a backfilled 2019 10-K is not a second code path with a second bug.

Workers recycle after on the order of 150 filings, and a supervisor kills a worker that has not heartbeated in minutes. SGML + lxml + a pathological table will leak; process restart is cheaper than a perfect allocator.

Cleaning markdown before it hits the indexCopy Anchor Link

Raw HTML→markdown from EDGAR is still dirty. A second pass:

• Re-parse pipe tables into cells, drop rows that are entirely empty, drop columns that are entirely empty.
• Tables with fewer than two remaining lines are deleted (a header with no body is not Item 8).
• Collapse three-or-more blank lines. 10-Ks are full of <BR><BR><BR>.
• Do not “prettify” numbers or rewrite captions. Cleaning is structural. If you start fixing “Net sales” spacing with a regex you will eventually eat a CUSIP.

Page markers (<PAGE> in older text filings) are counted so pages on the Document item is not always “how many PDF pages PyMuPDF saw.” HTML 10-Ks often have no useful page breaks; then pages is a lower bound.

Historical holes without a second parserCopy Anchor Link

The live Atom window is ~16 hours × business days. Everything else is missing-accession mode: download/extract submissions.zip (or a directory of CIK JSON already unpacked), collect accessions from a start date, diff against what we already stored, enqueue at lower priority than live. Live 8-Ks still jump the queue.

You can shard that inventory with a virtual worker id (id % N) so three machines do not parse the same 2018 10-K. You can skip chunking when the exhibit TYPEs we already stored are a superset of what the SGML contains — useful when you only needed to backfill XML extracts. The parse function is the same module as live. Two parsers is how historical 20-Fs silently lose ACCEPTANCE-DATETIME.

A done-log (append-only accessions) is the cheap checkpoint. If a worker dies mid-flush, the accession is not in the log and will be retried. Corrections in live still delete-by-accession even if the done-log has the old id.

What we persistCopy Anchor Link

Two families of writes, in parallel when both have work:

1. Filing + chunk documents — the markdown the product searches and returns. This is a document store, not a SQL table of paragraphs. The filing record nests documents[] (sequence, filename, chunk counts). Chunks are first-class rows.
2. Structured extracts — XML profiles, N-PORT-style holdings, XBRL facts from the IDEA zip. Those rows live in a columnar store. They are not a substitute for the markdown. See XBRL from the same filing.

We do not upload the raw .txt to object storage on the US hot path. The SEC already hosts it. We store our markdown and our facts. After a successful live flush we fire an internal “new document” signal so the rest of the product can wake up; that is not a public webhook.

On the read side, Document Contents loads chunks for one document_id, sorts by chunk_idx, and merges consecutive markdown. You never pass a chunk index as id. id is the file. A filing id (the accession) is rejected; take document_id from the filing, which is the primary file, or from documents[].

If you are building thisCopy Anchor Link

• Poll the Atom feed. Do not scrape the HTML “latest filings” page.
• Always fetch {accession}.txt. Treat HTML as an exhibit inside it.
• Prefer HTML to PDF; never index both.
• Write your own table-preserving HTML→markdown. Blog extractors will destroy Item 8.
• Chunk on headers and tables, not on token counts alone. Repeat table headers on overflow.
• Key exhibits separately from filings. Use integer CIK in Archives URLs and padded CIK on the company record.
• Share one rate limiter across poll and download.
• Re-parse on correction; do not patch.

The public Document item (document_id, filing_id, filed_at, source_document_url, source_filing_url) is just this pipeline, frozen. source_document_url is the SEC exhibit (…/data/{cik_int}/{accession_nodash}/{filename}); it is null when we never learned a filename. source_filing_url is the index. backlink_url is our viewer, which uses the padded CIK.

RelatedCopy Anchor Link