Translating EDINET filings at scale
How we pull Japan FSA EDINET packages, join the HTML, run quantized Japanese-to-English translation, and chunk them with the same pipeline as SEC filings.
EDINET is not EDGAR with different HTML. The Japan FSA ships ZIP packages of XHTML (often iXBRL), audit reports, and attachments, in Japanese, with form codes that do not look like 10-K. If you download one Yuho and paste it into a general chat model, you will wait, you will pay, and you will destroy tables.
We ingest EDINET continuously — historical catch-up from 2015-10-29 (the span on EDINET coverage), plus a live lookback over recent calendar days — and we translate at the HTML tree, with models kept small enough that a worker can hold them resident. The markdown chunker after that is the same header-aware chunker as US filings (Turning SEC filings into markdown). Search then sees source: edinet next to sec.
List by day, download the package
Discovery is the EDINET v2 list API: one calendar day at a time (date=YYYY-MM-DD, type=2). There is no Atom firehose. A producer walks days forward from a checkpoint, then in live mode re-scans a short lookback (several recent days) on a 1–2 minute sleep so late posts still land.
For each docID we pull:
type=1— the main document ZIPtype=3— the attached-docs ZIP when the list row says attachments exist
Dedup is on docID. An incomplete prior attempt (filing row without chunks) is deleted and retried. We do not treat “we saw this id once” as success.
Identifiers you will see later on Document items:
| Id | Meaning |
|---|---|
edinet_code | Filer in the FSA system. Accepted as company id on Entity Reference / Document Search. |
EDINET docID | Regulator submission id (source_filing_id in the store) |
| Our filing id | Stable internal id for the row we index (the public filing_id on EDINET documents) |
document_id | One exhibit inside that submission (main, audit, attachment) |
Company id on Document Search is still an Entity handle (ticker, CIK, or EDINET code). A US-only listing will not search EDINET; a Japan-only filer will not search SEC. XBRL in this API is US statement forms only — Japan tagged data is not merged into XBRL Search.
Join the ZIP into HTML documents
Inside the zip we prefer PublicDoc/ (and AuditDoc/ when present), else the XBRL/PublicDoc/ layout. All primary .htm / .html in the public tree are concatenated into one HTML document (styles merged). Audit reports stay separate documents. Attachments from AttachDoc/ become their own HTML files.
Encoding is messy: UTF-8 first, then detection, then cp932. If you assume UTF-8 you will mojibake a large minority of older packages.
We strip xmlns* noise and display:none nodes (iXBRL hidden labels). PDFs in the zip are skipped — they are a different pipeline and not worth the live translator. Images are not translated; they are stored in object storage and img src is rewritten to our asset URLs so the HTML still renders after we throw away the zip.
Filer display names prefer the alphabetic (Latin) submitter name from the EDINET directory when we have it. We do not spend model capacity re-translating “Toyota Jidosha K.K.” every filing.
Form codes are not 10-Ks (but we map them)
EDINET docTypeCode values are numeric (120, 140, 370, …). Internally we map them onto a small set of report / disclosure buckets so the rest of the product can filter:
- Annual securities reports (
120/130) sit with 10-K-like financial reports - Quarterly (
140/150) sit with 10-Q-like - Extraordinary reports, treasury, AGM materials, M&A packs have their own buckets
That map is a convenience for filing_type / disclosure filters. It is not a claim that a Yuho is a Form 10-K. The legal regime is different; the markdown is still Japanese-company prose, in English after translation.
Translation: small models, HTML in, HTML out
The bottleneck is Japanese body text, not download. A frontier LLM per exhibit does not meet a continuous ingest budget: too slow, too expensive, and too willing to “helpfully” rewrite tables.
We run specialized Japanese→English translation models, quantized (INT8) so each worker process keeps a replica in a few gigabytes of memory and does not pay a cold-load per filing. Throughput comes from many workers × short sequences, not from one giant context window.
What the worker actually does:
- Parse the joined HTML into a tree.
- Walk text nodes only. Tags, attributes,
href,src, table structure, and iXBRL element names stay put. If you serialize to plain text, translate, and hope, you will drop column alignment and you will translateix:nonNumericnames into poetry. - Batch sentences / cells into the model. Numbers, tickers, and Latin filer names are passed through. We seed a glossary from EDINET alphabetic names so issuers do not oscillate between three English spellings.
- Write the translated strings back into the same nodes. Re-serialize HTML.
- Quality gate: residual hiragana and katakana mean the model missed a span — retry that document (a handful of attempts). Kanji in proper nouns and some headings can remain. That is accepted. A gate that demands zero CJK will loop forever on names the glossary did not cover.
- After a bounded number of documents, recycle the worker and reload the quantized weights. Long-running interpreters fragment; reload is cheaper than a clever allocator.
We do not send filings to a general chat endpoint. We do not round-trip the HTML through a consumer translate widget. The model is a translation model. Speed is the product of quantization, batching, and preserving the tree so we never re-layout the document.
What we preserve on purpose: table grids, numeric cells, Latin tickers, EDINET codes, ix:* element names, image URLs (rewritten to our store). What we accept losing: honorifics, some legal boilerplate rhythm, occasional kanji in issuer names the glossary missed. What we refuse to do: summarize, skip “unimportant” tables, or translate into markdown first. Markdown is a later, lossy view. Translation has to happen on the tree the FSA shipped.
Filer-name translation for the directory (the alphabetic field when FSA did not supply Latin) is a separate, offline pass. The live filing path should not block on that.
Historical catch-up vs live lookback
EDINET’s list API is day-grained. That forces two producer modes:
Catch-up. Start at the indexed history start (2015-10-29 in the current coverage snapshot), write next_start_date after each successful day, cap how many filings a producer generation will take so a crash does not lose a month of cursor state. Incomplete days stay on the same next_start_date.
Live. Once caught up, re-list a rolling window of recent days (about five) on a 60–120 second sleep. Filers post late; a “today only” poll misses yesterday’s 16:00 Tokyo dump when your process was bouncing.
Download is not in the lister. The lister enqueues metadata batches (small — a handful of docs). Parser workers pull the ZIPs. That keeps HTTP/2 and FSA rate behavior in one place and lets us kill a stuck download without killing the day cursor.
If the document store already has a filing id for that docID and chunks, skip. If it has a filing and zero chunks, delete and retry — that is the translation-timeout case. Duplicates from overlapping live windows are cheaper to look up and skip than to unique in the FSA API (it will not).
What we store besides markdown
Translated HTML and images go to object storage under the internal filing id. Re-chunking a bad markdown run should not cost another model pass. Public image URLs are rewritten to our asset prefix so the HTML is usable without the zip.
The filing record carries EDINET docTypeCode / formCode / ordinanceCode as-is, plus our mapped filing_type, plus arrays of related edinet_codes (issuer, subject, subsidiary when the list row has them) and Japan secCodes. all_countries is JP. Industry codes from the filer directory map onto SIC-like buckets for filters that already speak SIC — that map is a convenience, not a regulator assignment.
filed_at is the EDINET submission clock as Japan calendar time, converted to a UTC instant with a Z. Do not apply the SEC “fake Z means Eastern” rule. Search from_date / to_date are submission dates in Japan, not a UTC-day cut of filed_at. Compact stamps look like yyyyMMdd:HHmm in the raw feed; we normalize.
Object storage holds translated HTML and images so we can re-chunk later without re-translating. The zip is discarded after a successful parse.
Then the same chunker as EDGAR
Translated HTML goes through the same HTML processor as EDGAR: tree cleanup → HTML→markdown → header-aware chunks (target 2048, cap ~10k, small overlap). Audit vs main vs attachment remain separate exhibits with their own document_ids.
Scale shape
Parsers and translators are separate process pools. Download is concurrent but bounded (small batches of list rows). The translator queue is the backpressure valve: if translation lags, the producer stops listing new days. Live mode is a rolling window, not “only today,” because EDINET often posts with a delay.
Japan filings access is entitlement-gated. Asking sources=edinet without that access is 403. Omitting sources searches every corpus the key can see.
If you are building this
- Use the list API by day; checkpoint
next_start_date. - Join PublicDoc HTML; keep audit/attachments as siblings.
- Translate the DOM, not innerText.
- Quantize a translation model; do not use a chat LLM as your ingest worker.
- Reuse one markdown chunker for SEC and EDINET or your search ranking will disagree.
- Keep integer vs padded ids off this corpus — EDINET codes are not CIKs.
- Expect leftover kanji. Design the gate around kana, not CJK.