文档索引
在以下地址获取完整的文档索引：https://docs.typesafe.ai/llms.txt 使用此文件可以在进一步探索之前发现所有可用页面。

使用置信度进行分类

将 SEC 年度报告分类到 75 个行业组中，每份报告使用一个 Choice，然后读取答案自身的置信度，以决定是上报该行业组还是其所属的更广泛的部门。

每家在 SEC 提交年度报告的公司都会在报告中描述其自身业务。我们将这些描述按照标准行业分类（Standard Industrial Classification）进行分类：75 个行业组，每份文档对应一个 Choice 问题。

大多数文件很容易分类。区域性银行就是区域性银行。但也有一些不易分类：例如一家刚刚出售了其两个分部之一的公司，或一家描述其计划进入而非正在运营的业务形式的初创公司。模型必须选择一个行业组，且困难案例的答案与简单案例的答案看起来并无二致。区分困难案例与简单案例通常就是成本所在：需要第二个模型、额外调用或人工审查。

Choice 已经告诉了你。除了胜出的选项外，它还会返回 confidence（置信度），当概率几乎全集中在一个选项上时置信度高，当概率分散在多个选项上时置信度低。这单一数值将可信答案与不可信答案区分开来。

如何处理不可信的答案取决于你的标签体系。SIC 标签形成层级结构：行业组归入更广泛的部门之下。这使得一次响应几乎无需额外成本。当模型对行业组不确定时，上报其所属的部门。由于宽泛标签由狭义标签推导而来，因此无需二次调用。

在 60 份文件中，0.9 的置信度阈值将其一分为二。高置信度的一半在 90% 的情况下是正确的；另一半仅为 40%。如果上报上一级标签，该 40% 会提升至 70%。最终我们得到一个 classify() 函数，每次请求处理一份文档，返回标签及其特异性程度。

flowchart LR
    doc["Item 1 'Business'<br/>from one 10-K"]

    subgraph request["one request"]
        q["Choice<br/>75 industry groups"]
    end

    sure{"confidence<br/>&ge; 0.9?"}
    grp["report the industry group<br/><i>e.g. 28</i>"]
    div["report its division<br/><i>e.g. manufacturing</i>"]

    doc --> request --> sure
    %% both branches leave the test, so they share a rank and stack on their own
    sure -- "yes" --> grp
    sure -- "no" --> div

Setup

pip install ipython matplotlib "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/

然后设置 TYPESAFE_API_KEY。每次 API 调用都缓存到 json_cache.json，该文件随 cookbook 一起提供，因此重新渲染会重现已发布的数字而不调用 API。删除该文件以重新实时运行所有内容。

以下数字来自 2026-08-12 的 jev-1.12。

import json
from collections import defaultdict
from pathlib import Path

import matplotlib
import matplotlib.pyplot as plt
from cooksafe import JsonCache, make_playground_link
from IPython.display import Markdown, display
from typesafe_sdk import Choice, TypeSafeClient

matplotlib.use("Agg")  # headless render

import os  # noqa: E402

TYPESAFE_MODEL = "jev-1.12"
CONFIDENT = 0.9  # above this the group is reported; below it, the division

client = TypeSafeClient(
    api_key=os.environ.get(
        "TYPESAFE_API_KEY", "cache-only"
    ),  # keyless kernels replay the cache
    base_url=os.environ.get("TYPESAFE_ENDPOINT"),
    timeout=120.0,
)
json_cache = JsonCache(Path("json_cache.json"))

构建分类的两个级别

sic_codes.tsv 是 SEC 发布的行业列表，申报者可从中选择自己的代码，获取于 2026-08-10：444 个四位代码，每个带有行业标题。数字是一个层次结构。前两位是 主要组（这里共有 75 个，从 01 农业生产到 99 不可分类），主要组的固定范围组成了十个部门，这是 SIC 最广泛的划分。

两个级别都来自该文件，没有涉及模型：按前两位数字对代码进行分组，然后将这些数字映射到部门。

DIVISIONS = [
    (1, 9, "agriculture, forestry and fishing"),
    (10, 14, "mining"),
    (15, 17, "construction"),
    (20, 39, "manufacturing"),
    (40, 49, "transportation, communications and utilities"),
    (50, 51, "wholesale trade"),
    (52, 59, "retail trade"),
    (60, 67, "finance, insurance and real estate"),
    (70, 89, "services"),
    (91, 99, "public administration"),
]

INDUSTRIES: dict[str, str] = {}
for line in Path("sic_codes.tsv").read_text().splitlines()[1:]:
    code, _office, title = line.split("\t")
    INDUSTRIES[code] = title.lower()

GROUPS: dict[str, list[str]] = defaultdict(list)
for code in sorted(INDUSTRIES):
    GROUPS[code[:2]].append(code)


def division(group: str) -> str:
    number = int(group)
    return next(name for low, high, name in DIVISIONS if low <= number <= high)


print(
    f"{len(INDUSTRIES)} industries -> {len(GROUPS)} major groups -> {len(DIVISIONS)} divisions"
)
print(
    f"  group 35 = {division('35')} / {', '.join(INDUSTRIES[c] for c in GROUPS['35'][:3])} ..."
)

444 industries -> 75 major groups -> 10 divisions
  group 35 = manufacturing / engines & turbines, farm machinery & equipment, lawn & garden tractors & home lawn & gardens equip ...

Choice 问题需要一个东西来描述每个选项，而组自己的名称并不总是存在：75 个中有 42 个在 SEC 的列表中带有伞式标题，其余没有。因此，每个组都通过其内部的行业来描述，这也是阅读申报文件的人将要匹配的内容。

MAX_NAMED = (
    8  # industries listed per group; enough to characterise it without a wall of text
)


def describe(group: str) -> str:
    umbrella = INDUSTRIES.get(f"{group}00")
    inside = [INDUSTRIES[c] for c in GROUPS[group] if c != f"{group}00"][:MAX_NAMED]
    listed = "; ".join(inside)
    return (
        f"{umbrella} — includes: {listed}"
        if umbrella and listed
        else (umbrella or listed)
    )


print(f"group 20: {describe('20')[:150]}")
print(f"\ngroup 65: {describe('65')[:150]}")

group 20: food and kindred products — includes: meat packing plants; sausages & other prepared meat products; poultry slaughtering and processing; dairy product

group 65: real estate — includes: real estate operators (no developers) & lessors; operators of nonresidential buildings; operators of apartment buildings; less

The filings

filings.jsonl 包含 60 份年度报告（10-K），每份均精简至第 1 项“业务”（Item 1 "Business"），即公司描述其业务内容的章节，这也是行业代码所涵盖的唯一部分。这些文件时间跨度为 1993–2024 年，字数从 700 到 2,200 不等。每份文件都携带提交人选择的 SIC 代码，以及用于在 EDGAR 上查找的访问编号。

在讨论任何准确率数字之前，该标签的来源至关重要。标签是公司自行上报的：提交文件的填报人一次性选定，当公司出售了该代码对应的业务但仍保留该代码时，标签就会过期。这 60 份文件经过筛选，仅保留其正文内容支持其所附代码的文件，因此此处的数据衡量的是判定逻辑，而非 EDGAR 元数据的现状。

FILINGS = [json.loads(line) for line in Path("filings.jsonl").read_text().splitlines()]
example = FILINGS[7]
print(
    f"{len(FILINGS)} filings, {sum(f['words'] for f in FILINGS) // len(FILINGS)} words on average"
)
print(f"\n{example['id']} (filed {example['year']}, accession {example['accession']}):")
print(f"  {example['text'][:230]}...")
print(f"  filer's code: {example['sic']} {INDUSTRIES[example['sic']]}")

60 filings, 1438 words on average

1389870_2008 (filed 2008, accession 0001079974-09-000155):
  Item 1. DESCRIPTION OF BUSINESS. NARRATIVE DESCRIPTION OF THE BUSINESS Across America Financial Services, Inc. is a corporation which was formed under the laws of the State of Colorado on December 1, 2005. Until March 23, 2007, we...
  filer's code: 6163 loan brokers

询问一个 Choice 问题，并读取置信度

一个 Choice 问题，其选项为 75 个行业组。整个分类法可容纳在一个请求中：Choice 最多可可靠处理约 240 个选项，而 75 远低于此限制。

The answer comes back with choice, the winning group; probabilities, the weight on each of the 75; and confidence, which says how concentrated that spread was. The recipe reads confidence rather than the winner's own probability. A winner at 0.45 with a runner-up at 0.44, and a winner at 0.45 with the rest of the weight scattered thinly, are different situations, and confidence is what separates them.

QUESTION = (
    "Which broad industry does this company operate in? Judge the company's own operations "
    "as this filing describes them."
)


def questions() -> dict:
    return {
        "group": Choice(
            instructions=QUESTION,
            criteria={group: describe(group) for group in sorted(GROUPS)},
        )
    }


@json_cache
def ask(filing_id: str, text: str) -> dict:
    response = client.system_one(
        state=text, questions=questions(), model=TYPESAFE_MODEL
    )
    answer = response.answers["group"]
    return {
        "group": answer.choice,
        "confidence": answer.confidence,
        "probabilities": dict(answer.probabilities),
    }

有把握时返回组，没把握时返回其部门

下面四行是整个配方。在 0.9 置信度或更高时，答案报告为行业组；低于该值时，同一答案报告为该组所在的部门。

每份文件都会返回一个可用的标签。对于模型无法自信地分类的文件，会返回上一级标签，而非丢弃或进一步处理。如果部门标签对你的应用来说过于粗略，这里是将其转交人工处理的分支。

def classify(filing: dict) -> dict:
    answer = ask(filing["id"], filing["text"])
    sure = answer["confidence"] >= CONFIDENT
    return {
        "level": "group" if sure else "division",
        "label": answer["group"] if sure else division(answer["group"]),
        "confidence": answer["confidence"],
        "group": answer["group"],
    }


def show(filing: dict) -> None:
    result = classify(filing)
    named = describe(result["group"]).split(" — ")[0][:46]
    print(
        f"  {filing['id']:>13}  conf {result['confidence']:.2f}  -> {result['level']:<8} "
        f"{result['label']:<14} (group {result['group']}: {named})"
    )


print("three filings the model was sure about:")
for f in sorted(FILINGS, key=lambda f: -ask(f["id"], f["text"])["confidence"])[:3]:
    show(f)
print("\nthree it was not:")
for f in sorted(FILINGS, key=lambda f: ask(f["id"], f["text"])["confidence"])[:3]:
    show(f)

three filings the model was sure about:
    310158_1996  conf 1.00  -> group    28             (group 28: chemicals & allied products)
     33416_1998  conf 1.00  -> group    63             (group 63: life insurance; accident & health insurance; h)
    352541_1996  conf 1.00  -> group    49             (group 49: electric, gas & sanitary services)

three it was not:
   1372167_2013  conf 0.22  -> division manufacturing  (group 38: search, detection, navagation, guidance, aeron)
   1398633_2009  conf 0.23  -> division wholesale trade (group 50: wholesale-durable goods)
     46653_1999  conf 0.29  -> division services       (group 87: services-engineering, accounting, research, ma)

置信度与各份文件分类的难度相符。置信度为 1.00 的三份文件分别是一家制药商、一家生命保险公司和一家公用事业公司；这三家公司在账面上均为控股公司，但每家公司都在文件中明确指出了其一项主导业务。置信度最低的三份文件之所以难分类，原因可在文本中找到。其中两家是初创公司，描述的是它们计划开展的业务（Nevaeh“计划作为软件开发者运营”，Barricode“旨在进入计算机安全软件行业”），第三家则有两个业务分部，并在提交文件前几周出售了其中一个。这三份文件最终被分类为部门（division）而非行业组（group）。

classify() 是整个配方。将 ask() 指向你自己的文档，并为你的分类法重写 describe()，其余部分可以复用。

宽泛答案带来的价值

所有 60 份申报文件，根据每个申报者选择的代码进行评估，在两种政策下：每次命名一个组，或者当置信度低于 0.9 时报告部门。

def correct(filing: dict, result: dict) -> bool:
    gold_group = filing["sic"][:2]
    if result["level"] == "group":
        return result["label"] == gold_group
    return result["label"] == division(gold_group)


results = [(f, classify(f)) for f in FILINGS]
sure = [(f, r) for f, r in results if r["level"] == "group"]
unsure = [(f, r) for f, r in results if r["level"] == "division"]

forced = sum(r["group"] == f["sic"][:2] for f, r in results)
broadened = sum(correct(f, r) for f, r in results)

print(f"forced to name a group every time      {forced}/{len(results)} right")
print(
    f"  of those, the {len(sure)} it was sure about  "
    f"{sum(r['group'] == f['sic'][:2] for f, r in sure)}/{len(sure)} right"
)
print(
    f"  and the {len(unsure)} it was not           "
    f"{sum(r['group'] == f['sic'][:2] for f, r in unsure)}/{len(unsure)} right"
)
print(
    f"\nletting it answer coarsely when unsure  {broadened}/{len(results)} useful answers"
)

forced to name a group every time      39/60 right
  of those, the 30 it was sure about  27/30 right
  and the 30 it was not           12/30 right

letting it answer coarsely when unsure  48/60 useful answers

在模型确定的情况下，其命名的行业组有九成是正确的。在模型不确定的情况下，命名行业组错误的次数多于正确，准确率为 40%。将这些答案上报为部门，准确率将提升至 70%。

该图表将两种策略并列展示，按模型是否确定进行划分。

labels = ["sure\n(group reported)", "unsure\n(division reported)"]
forced_split = [
    sum(r["group"] == f["sic"][:2] for f, r in sure) / len(sure),
    sum(r["group"] == f["sic"][:2] for f, r in unsure) / len(unsure),
]
broad_split = [
    sum(correct(f, r) for f, r in sure) / len(sure),
    sum(correct(f, r) for f, r in unsure) / len(unsure),
]

fig, ax = plt.subplots(figsize=(7, 3.6))
x = range(len(labels))
ax.bar(
    [i - 0.19 for i in x],
    forced_split,
    0.38,
    label="always name a group",
    color="#c8ccd4",
)
ax.bar(
    [i + 0.19 for i in x],
    broad_split,
    0.38,
    label="answer broadly when unsure",
    color="#3b6ea5",
)
for i, (a, b) in enumerate(zip(forced_split, broad_split)):
    ax.text(i - 0.19, a + 0.02, f"{a:.0%}", ha="center", fontsize=9)
    ax.text(i + 0.19, b + 0.02, f"{b:.0%}", ha="center", fontsize=9)
ax.set_xticks(list(x))
ax.set_xticklabels(
    [f"{lab}\nn={n}" for lab, n in zip(labels, [len(sure), len(unsure)])]
)
ax.set_ylabel("labels that are right")
ax.set_ylim(0, 1.12)
ax.set_title("Where the broader answer helps: the filings it was unsure about")
ax.legend(frameon=False, loc="upper right")
ax.spines[["top", "right"]].set_visible(False)
plt.tight_layout()
display(fig)

Open it in the playground

此共享链接包含一个申报文件和包含 75 个选项的问题，因此你可以在不编写任何代码的情况下看到其分布和产生的置信度。

playground_link = make_playground_link(
    example["text"], questions(), models=[TYPESAFE_MODEL]
)
display(
    Markdown(
        f"🔗 [Open the filing + question in the TypeSafe playground]({playground_link})"
    )
)

Open the filing + question in the TypeSafe playground →