Topaz 5.21.1

One wayto say it.

Topaz is a general-purpose programming language honed first for typesetting books and documents. Code that does the same job looks the same whoever writes it, so code you have never seen reads quickly.

TOPAZ
import std.io

let labels = ["12", "cover", "14"]
let pages = labels.map(text => text.toInt())
io.writeLine("{pages}")
output
[Some(12), None, Some(14)]

map goes after the list and toInt after each string. The entry for “cover” becomes None.

Who, for what

Built for these teams

Teams building a book typesetting engine

From manuscript to pages

Build an engine that takes a manuscript and decides where lines and pages fall. Paragraph breaking and the Korean, Japanese and Chinese line-break rules come from the standard library. The same line-breaking code runs in the browser preview and in the print composition.

Browser package · Executable

Teams whose rules differ by publisher

Per-book decisions in rule files

Where a chapter starts and how many orphan lines to allow differ by publisher and by book. Those decisions live in Lispex rule files, and the line-breaking code is shared across books. When each client wants its own style for a textbook or a report, the app code stays the same.

Executable · Lispex rules

Teams building web tools

Data handling inside a web app

Write a function that handles input, such as checking JSON, in Topaz and build it as a browser package. Your existing web app calls it from JavaScript. The browser tools on jeni.kr work this way.

npm · Browser package

std.typeset

Typesetting in the standard library

std.typeset in Topaz 5.21 is the standard module that breaks paragraphs into lines. The program below sets one Korean sentence at a width of ten Hangul syllables. Looking at the whole paragraph keeps a lone word off the last line.

TOPAZ
import std.io
import std.typeset {
  BreakCluster, BreakPolicy, BreakMode, KinsokuRules, BreakOpportunity, Paragraph, Item,
  Box, Penalty, BreakRule, DistanceGrid, CostModel, LineGeometry, JustificationPolicy,
  JustificationMode, KnuthPlassPolicy, ParagraphPlan, PlanLine, classifyBreaks,
  breakParagraph,
}

let sentence = "문단 전체를 보고 줄을 나누면 마지막 줄이 짧게 남지 않고 줄마다 길이가 고르게 됩니다."
let clusters = sentence.scalars().map(cluster)
let result = classifyBreaks(clusters, koreanRules(clusters))
    .flatMap((breaks: Array<BreakOpportunity>) => breakParagraph(paragraph(clusters, breaks)))
match result {
    case Ok(plans) => {
        io.writeLine("Whole paragraph\n{linesOf(clusters, plans.optimized)}\n")
        io.writeLine("Line by line\n{linesOf(clusters, plans.greedy)}")
    }
    case Err(error) => io.writeLine("{error}")
}

The main part of the program. The full source adds the helpers that set character widths and build the paragraph.

output
Whole paragraph
문단 전체를 보고
줄을 나누면 마지막
줄이 짧게 남지
않고 줄마다 길이가
고르게 됩니다.

Line by line
문단 전체를 보고 줄을
나누면 마지막 줄이
짧게 남지 않고 줄마다
길이가 고르게
됩니다.

Paragraph breaking

Get the breaks chosen for the whole paragraph together with a greedy, line-by-line result. Choose the Knuth-Plass cost model or a fixed-point cost model.

Unicode line breaking

Find break opportunities with the Unicode 16.0 line breaking algorithm (UAX #14).

Rules for each locale

Add the Korean (KLReq), Japanese (JLReq), or Chinese (CLReq) rules, and an opening bracket never ends a line while a closing bracket or period never starts one. The Japanese rules follow the JLReq character classes.

East Asian width

The Unicode East Asian Width property decides breaks around quotation marks and brackets next to CJK text.

Spacing

Set how far justification may stretch or shrink the space between Latin words, and how far it may stretch the space between CJK characters.

Length units

Write lengths such as 12pt, 3mm, 1cm, 1in, and 1em directly in code and convert between units.

ATLAS

A typesetting tool built with Topaz

ATLAS turns a manuscript into print-ready book pages. Here is a page it set, with part of its line-breaking rules in Topaz. ATLAS names these functions with Korean typesetting terms, shown here in English.

Page 18 of a Korean book set with ATLAS
Page 18 of a book set with ATLAS
TOPAZ
function nextIsNotLatinSpace(next: Option<string>) -> bool {
  match next {
    case Some(text) => !isLatinSpace(text)
    case None => true
  }
}
function nextHasNoLineStartBan(next: Option<string>) -> bool {
  match next {
    case Some(text) => !hasLineStartBan(text)
    case None => true
  }
}

Part of the ATLAS line-breaking rules, with names translated into English for illustration

Why Topaz handles typesetting well

One way to say it

Common tasks have one way to write them. Whether a person or an agent writes the code, it takes the same shape, so a review only has to follow what the code does.

One way to write it

String conversion and collection operations are methods on the value. Since 5.21 there is no global function form for them, and topaz fmt is the one formatter.

Moving to 5.21 →

Missing and failed are different values

A value that may be missing is an Option, an operation that may fail returns a Result, and cleanup is written with defer. At the end of a paragraph there is no next character, so a line-break rule receives None rather than an error.

Null, Option, Result & Faults →

The outside world shows up in imports

Printing to the screen and reading files both use a standard module you import. A reviewer finds every point of input and output from the imports.

Standard Library Overview →

Missing cases stop the build

Put a fixed set of cases, such as line-break classes, in an enum, and the compiler checks that each match handles every one. When you add a case later, the build reports each match that does not handle it yet.

Patterns & Control Flow →

Overflow stops on the spot

An int is a signed 64-bit integer, and overflow stops the program at that point instead of continuing with a wrong value. When you measure line widths and layout costs in integers, an overflow cannot hide in the result.

Runtime Behavior →

The same rules on every target

Direct runs, native executables and browser packages follow the same language rules. So the on-screen preview and the print layout break lines at the same places.

Execution Targets →

Lispex

Topaz computes, Lispex decides

Computing such as line breaking happens in Topaz code. Decisions that change with each publisher and book, such as which page a chapter starts on or how many lines may stand alone, go into separate Lispex rules. The Lispex 1.21 evaluator built into Topaz 5.21 runs those rules within the limits set in a limits file. Change a rule file and lock the package again, and the same manuscript takes another style without any change to the application code.

  • Rules never touch files, the system clock, or the network.
  • A rule stops where it runs past its limit.
  • The same rule, input, and settings give the same decision every time.
LISPEX
(let ((min-lines 2))
  (if (< input min-lines) "next-page" "keep"))

All of rules/widow-lines.lspx

TOPAZ
import std.io
import src.values { singleDigit, textOf }
import std.lispex { defaultLimits, evaluate }
import std.lispex.rules as rules

export function main(args: Array<string>, stdin: string) -> Result<int, string> {
    let rule = rules.widow_lines()
    for linesLeft in [1, 2] {
        match evaluate(rule, singleDigit(linesLeft)?, defaultLimits(rule)) {
            case Ok(Complete(result)) => io.writeLine("lines left {linesLeft}: {textOf(result)}")
            case _ => return Err("the rule did not finish")
        }
    }
    Ok(0)
}
output
lines left 1: next-page
lines left 2: keep

All of src/main.tpz. singleDigit and textOf are small helpers in src/values.tpz that convert Lispex values.

Lispex rules run in the interpreter and in native builds.

Built with Topaz

Beyond typesetting

The core of the Topaz compiler is written in Topaz. Browser tools are written in the same language.

TOPAZ
function assignmentRoot(node: SyntaxNode) -> Option<SyntaxNode> {
  if node.kind == "expression/identifier" {
    return Some(node)
  }
  match child(node, "object") {
    case Some(object) => assignmentRoot(object)
    case None => None
  }
}
An excerpt from compiler/crates/topaz_self_frontend/topaz/checker/expression_core.tpz published in studiohaze/topaz

The compiler

Code written in Topaz checks names and types and generates the Rust. Every release ships that source as one seed archive.

Built with Topaz →

Browser tools

The JSON formatter on jeni.kr tidies input with a parser written in Topaz, and reports the line and column when a closing bracket is missing.

Call it from JavaScript →