Typesetting

Use std.typeset to break a paragraph into lines of a given width and to work out how far each line's spaces stretch or shrink. Paragraphs that mix Korean, Chinese, Japanese, and Latin text break by the Unicode line breaking rules and the kinsoku rules of each language. Kinsoku rules keep characters such as closing brackets and full stops from starting a line, and characters such as opening brackets from ending one.

Import the module with import std.typeset in the 5.21 language mode. Install Topaz through First Program to run the examples. Find the other standard modules in Standard Library Overview.

Break one paragraph into lines

Save this as typeset-paragraph.tpz. Widths are in millipoints, a thousandth of a point. Assuming a 10pt font, characters at code point 4352 (U+1100) and above are 10,000 wide, other characters 5,000, and the space 2,500. The line width is 100,000, or 100pt.

TOPAZ

import std.io
import std.typeset

function advance(ch: string) -> int {
  if ch == " " { return 2500 }
  match ch.codePointAt(0) {
    case Some(code) => if code >= 4352 { 10000 } else { 5000 };
    case None => 0
  }
}

function layout(text: string, width: int) -> Result<Array<string>, typeset.TypesetError> {
  let chars = text.scalars()
  let clusters = chars.map(ch => typeset.BreakCluster { text: ch, advance: advance(ch), trackingAfter: 0 })
  let points = typeset.classifyBreaks(clusters, typeset.BreakPolicy {
    mode: typeset.BreakMode.Word, rules: typeset.KinsokuRules.Unicode16Ko,
    forbidBefore: [], forbidAfter: [], boundaryAllowed: chars.map(ch => true),
    generatedHyphenEnds: [], protectFittingLatin: false, preserveShortWords: false,
    narrowestWordTarget: width, bodyWordTarget: width
  })?

  let mut items: Array<typeset.Item> = []
  for i in 0..<chars.length {
    items.push(typeset.Item.BoxItem(typeset.Box { unitIndex: i, text: chars[i], width: advance(chars[i]), trackingAfter: 0 }))
  }
  for p in points {
    let rule = if p.kind == typeset.BreakKind.ForcedEnd { typeset.BreakRule.Forced } else { typeset.BreakRule.Allowed(0) }
    items.push(typeset.Item.PenaltyItem(typeset.Penalty { unitEnd: p.unitEnd, width: 0, rule: rule, kind: p.kind, flagged: false }))
  }

  let result = typeset.breakParagraph(typeset.Paragraph {
    grid: typeset.DistanceGrid.MilliPoint,
    items: items,
    geometry: typeset.LineGeometry { bodyTarget: width, firstTarget: width, dropCapSpan: 0, dropCapInset: 0 },
    justification: typeset.JustificationPolicy {
      mode: typeset.JustificationMode.Unjustified, minGapCount: 0, minLineWidthRatioM: 0,
      minFinalLineWidthRatioM: None, maxAdjustment: 0, maxGapAdjustment: 0, maxGapRatioR: 0,
      latinSpacing: None, cjkStretchRatio: 0.0
    },
    costModel: typeset.CostModel.FixedPoint,
    knuthPlass: typeset.KnuthPlassPolicy { toleranceR: 200000000, linePenalty: 10, fitnessDemerit: 10000, flaggedDemerit: 10000, finalFlaggedDemerit: 5000 }
  })?
  Ok(result.optimized.lines.map(line => chars.slice(line.start, line.trimmedEndExclusive).join("")))
}

let text = "토파즈는 책과 문서를 조판하는 언어입니다. 한국어와 English가 섞인 문단도 규칙대로 나눕니다."
match layout(text, 100000) {
  case Ok(lines) => { for line in lines { io.writeLine(line) } };
  case Err(error) => io.writeLine("{error}")
}

Run topaz run typeset-paragraph.tpz in the folder containing that file.

Output

토파즈는 책과 문서를
조판하는 언어입니다.
한국어와 English가
섞인 문단도
규칙대로 나눕니다.

The program breaks lines in three steps.

classifyBreaks decides, for each cluster (BreakCluster), whether a line may break after it. It returns a list of break opportunities (BreakOpportunity).
The program adds a Box for each character and a Penalty for each opportunity to a Paragraph. The ForcedEnd at the end of the paragraph is a mandatory break, so it becomes BreakRule.Forced.
breakParagraph plans the lines. Each PlanLine shows the units from start up to, but not including, trimmedEndExclusive.

BreakMode.Word keeps a Korean word together when it fits the line, so lines change only at word boundaries. breakParagraph returns an optimized plan optimized and a greedy plan greedy together. The section on greedy and optimal line breaking below compares them.

Lengths and units

Write typesetting dimensions as length values such as 12pt, 3mm, 1cm, 1in, and 1em. The unit is part of the type, so 148mm is a Length<Mm> and 10pt is a Length<Pt>. Only lengths with the same unit add and subtract. The checker rejects a mixed expression such as 90mm + 10pt with TPZ5001, so convert first.

TOPAZ

import std.io
import std.typeset

let page = 148mm
let margin = 20mm
let column = page - margin * 2
let fontSize = 10pt

io.writeLine("{column}")
io.writeLine("{column.toPt().toMilliPoints()}")
io.writeLine("{1em.resolve(fontSize).toMilliPoints()}")
io.writeLine("{typeset.fromMilliPoints(10000) == fontSize}")

Output

Length<Mm>(20063304)
306142
10000
true

148mm - 20mm * 2 is a Length<Mm>. The output Length<Mm>(20063304) shows the stored value in scaled points. One point is 65,536 scaled points.
toPt() expresses the same length in points, and toMilliPoints() returns the nearest whole number of millipoints. 108mm is 306,142 millipoints.
1em is a length relative to the font size. resolve turns it into points for a given font size.
typeset.fromMilliPoints and typeset.fromScaledPoints turn an integer back into a Length<Pt>.

Widths in the paragraph API are integers. Paragraph.grid names the unit of those integers. DistanceGrid.MilliPoint means millipoints and DistanceGrid.ScaledPoint means scaled points. CostModel.FixedPoint accepts only MilliPoint.

Measure glyph widths

std.typeset opens no fonts and reads no files. Your program measures each glyph with its font and passes the result as BreakCluster.advance and Box.width. If you add tracking, include the tracking after the glyph in the width and also record it in trackingAfter.

TOPAZ

import std.io
import std.typeset

let items: Array<typeset.Item> = [
  typeset.Item.BoxItem(typeset.Box { unitIndex: 0, text: "가", width: 10200, trackingAfter: 200 }),
  typeset.Item.BoxItem(typeset.Box { unitIndex: 1, text: "나", width: 10200, trackingAfter: 200 }),
  typeset.Item.BoxItem(typeset.Box { unitIndex: 2, text: " ", width: 2700, trackingAfter: 200 }),
  typeset.Item.PenaltyItem(typeset.Penalty { unitEnd: 3, width: 0, rule: typeset.BreakRule.Forced, kind: typeset.BreakKind.ForcedEnd, flagged: false })
]
let paragraph = typeset.Paragraph {
  grid: typeset.DistanceGrid.MilliPoint,
  items: items,
  geometry: typeset.LineGeometry { bodyTarget: 100000, firstTarget: 100000, dropCapSpan: 0, dropCapInset: 0 },
  justification: typeset.JustificationPolicy {
    mode: typeset.JustificationMode.Unjustified, minGapCount: 0, minLineWidthRatioM: 0,
    minFinalLineWidthRatioM: None, maxAdjustment: 0, maxGapAdjustment: 0, maxGapRatioR: 0,
    latinSpacing: None, cjkStretchRatio: 0.0
  },
  costModel: typeset.CostModel.FixedPoint,
  knuthPlass: typeset.KnuthPlassPolicy { toleranceR: 200000000, linePenalty: 10, fitnessDemerit: 10000, flaggedDemerit: 10000, finalFlaggedDemerit: 5000 }
}

match typeset.breakParagraph(paragraph) {
  case Ok(result) => {
    let line = result.optimized.lines[0]
    io.writeLine("raw {line.endExclusive} {line.rawTextWidth}")
    io.writeLine("trimmed {line.trimmedEndExclusive} {line.trimmedTextWidth}")
    io.writeLine("scalars {line.displayTextScalarLength}")
  };
  case Err(error) => io.writeLine("{error}")
}

Output

raw 3 22900
trimmed 2 20200
scalars 2

A line's width subtracts the tracking after its last unit once. rawTextWidth includes the trailing space, so it is 10,200 + 10,200 + 2,700 − 200 = 22,900. trimmedTextWidth leaves out the trailing space and the tracking after 나, giving 20,200. trimmedEndExclusive is position 2, before the space, and displayTextScalarLength counts the 2 scalars left on screen. Trailing spaces are U+0020 SPACE and U+00A0 NO-BREAK SPACE.

Break opportunities (UAX #14)

UAX #14 is the Unicode line breaking algorithm. It gives each character a line break class and decides whether a line may break between two characters. classifyBreaks applies the Unicode 16.0.0 data and rules LB1 to LB31 of UAX #14 revision 53, then adds the locale's kinsoku rules. The next example marks each break opportunity with |.

TOPAZ

import std.io
import std.typeset

function marked(text: string, rules: typeset.KinsokuRules, mode: typeset.BreakMode) -> string {
  let chars = text.scalars()
  let clusters = chars.map(ch => typeset.BreakCluster { text: ch, advance: 10000, trackingAfter: 0 })
  let policy = typeset.BreakPolicy {
    mode: mode, rules: rules, forbidBefore: [], forbidAfter: [],
    boundaryAllowed: chars.map(ch => true), generatedHyphenEnds: [],
    protectFittingLatin: false, preserveShortWords: false,
    narrowestWordTarget: 100000, bodyWordTarget: 100000
  }
  match typeset.classifyBreaks(clusters, policy) {
    case Ok(points) => {
      let ends = points.map(p => p.unitEnd)
      let mut out = ""
      for i in 0..<chars.length {
        out = "{out}{chars[i]}"
        if i + 1 < chars.length && ends.indexOf(i + 1) != None { out = "{out}|" }
      }
      out
    };
    case Err(error) => "{error}"
  }
}

let ko = typeset.KinsokuRules.Unicode16Ko
for text in ["Hello, world! It's 3.14.", "e.g. well-known", "equals .35 cents", "조판은 (보기처럼) 글자를 나눕니다."] {
  io.writeLine(marked(text, ko, typeset.BreakMode.Character))
}
io.writeLine(marked("조판은 (보기처럼) 글자를 나눕니다.", ko, typeset.BreakMode.Word))

Output

Hello, |world! |It's |3.14.
e.g. |well-|known
equals .35 |cents
조|판|은 |(보|기|처|럼) |글|자|를 |나|눕|니|다.
조판은 |(보기처럼) |글자를 |나눕니다.

A line may break after a space and after a hyphen. It does not break inside a number such as 3.14, before closing punctuation, or after an opening bracket.
UAX #14 alone allows a break before .35. Unicode16Ko never starts a line with a full stop, so it removes that opportunity.
BreakMode.Character may break between any two Hangul syllables. BreakMode.Word keeps a Korean word together when its width does not exceed bodyWordTarget.

Each opportunity carries a BreakKind. A break after a space is LatinSpace, after a hyphen LatinHyphen, after closing punctuation ClosingPunctuation, and after an East Asian character KoreanCluster. The end of the paragraph and a break after a line break character are a mandatory ForcedEnd. Any other position is HardFallback. The FixedPoint cost model adds 50,000B to a line broken there, where B is 1,000,000. The KnuthPlass model does not use the break kind in its cost.

East Asian characters and JLREQ classes

Hangul, Han characters, and kana are wide East Asian characters. A line may break between them even without a space. classifyBreaks treats characters whose Unicode East_Asian_Width is Fullwidth (F), Wide (W), or Halfwidth (H) as East Asian.

JLREQ is the W3C document Requirements for Japanese Text Layout. It sorts punctuation and kana into classes such as opening brackets, closing brackets, and small kana, and states for each class whether it may start or end a line. Unicode16Ja uses the strict kinsoku of JLREQ 3.1.7 to 3.1.10 and the class tables of Appendix A.

TOPAZ

import std.io
import std.typeset

function marked(text: string, rules: typeset.KinsokuRules, mode: typeset.BreakMode) -> string {
  let chars = text.scalars()
  let clusters = chars.map(ch => typeset.BreakCluster { text: ch, advance: 10000, trackingAfter: 0 })
  let policy = typeset.BreakPolicy {
    mode: mode, rules: rules, forbidBefore: [], forbidAfter: [],
    boundaryAllowed: chars.map(ch => true), generatedHyphenEnds: [],
    protectFittingLatin: false, preserveShortWords: false,
    narrowestWordTarget: 100000, bodyWordTarget: 100000
  }
  match typeset.classifyBreaks(clusters, policy) {
    case Ok(points) => {
      let ends = points.map(p => p.unitEnd)
      let mut out = ""
      for i in 0..<chars.length {
        out = "{out}{chars[i]}"
        if i + 1 < chars.length && ends.indexOf(i + 1) != None { out = "{out}|" }
      }
      out
    };
    case Err(error) => "{error}"
  }
}

let character = typeset.BreakMode.Character
io.writeLine("ko {marked("가나", typeset.KinsokuRules.Unicode16Ko, character)}")
io.writeLine("zh {marked("甲，乙", typeset.KinsokuRules.Unicode16Zh, character)}")
io.writeLine("zh {marked("甲（乙）", typeset.KinsokuRules.Unicode16Zh, character)}")
for text in ["カァナ", "カーキ", "あゝい"] {
  io.writeLine("ja {marked(text, typeset.KinsokuRules.Unicode16Ja, character)}")
}

Output

ko 가|나
zh 甲，|乙
zh 甲|（乙）
ja カァ|ナ
ja カー|キ
ja あゝ|い

A line may break between two Hangul syllables, as in 가|나.
The comma ， and the closing bracket ） never start a line, and the opening bracket （ never ends one.
The small kana ァ, the prolonged sound mark ー, and the iteration mark ゝ never start a line.

Greedy and optimal line breaking

Greedy line breaking fills one line at a time. It puts as much as fits on the current line and moves on without looking at later lines. Optimal line breaking compares combinations of break positions across the whole paragraph and picks the plan with the lowest total cost.

TOPAZ

import std.io
import std.typeset

function advance(ch: string) -> int {
  if ch == " " { return 2500 }
  match ch.codePointAt(0) {
    case Some(code) => if code >= 4352 { 10000 } else { 5000 };
    case None => 0
  }
}

function plan(chars: Array<string>, width: int) -> Result<typeset.ParagraphResult, typeset.TypesetError> {
  let clusters = chars.map(ch => typeset.BreakCluster { text: ch, advance: advance(ch), trackingAfter: 0 })
  let points = typeset.classifyBreaks(clusters, typeset.BreakPolicy {
    mode: typeset.BreakMode.Word, rules: typeset.KinsokuRules.Unicode16Ko,
    forbidBefore: [], forbidAfter: [], boundaryAllowed: chars.map(ch => true),
    generatedHyphenEnds: [], protectFittingLatin: false, preserveShortWords: false,
    narrowestWordTarget: width, bodyWordTarget: width
  })?

  let mut items: Array<typeset.Item> = []
  for i in 0..<chars.length {
    items.push(typeset.Item.BoxItem(typeset.Box { unitIndex: i, text: chars[i], width: advance(chars[i]), trackingAfter: 0 }))
  }
  for p in points {
    let rule = if p.kind == typeset.BreakKind.ForcedEnd { typeset.BreakRule.Forced } else { typeset.BreakRule.Allowed(0) }
    items.push(typeset.Item.PenaltyItem(typeset.Penalty { unitEnd: p.unitEnd, width: 0, rule: rule, kind: p.kind, flagged: false }))
  }

  typeset.breakParagraph(typeset.Paragraph {
    grid: typeset.DistanceGrid.MilliPoint,
    items: items,
    geometry: typeset.LineGeometry { bodyTarget: width, firstTarget: width, dropCapSpan: 0, dropCapInset: 0 },
    justification: typeset.JustificationPolicy {
      mode: typeset.JustificationMode.Unjustified, minGapCount: 0, minLineWidthRatioM: 0,
      minFinalLineWidthRatioM: None, maxAdjustment: 0, maxGapAdjustment: 0, maxGapRatioR: 0,
      latinSpacing: None, cjkStretchRatio: 0.0
    },
    costModel: typeset.CostModel.FixedPoint,
    knuthPlass: typeset.KnuthPlassPolicy { toleranceR: 200000000, linePenalty: 10, fitnessDemerit: 10000, flaggedDemerit: 10000, finalFlaggedDemerit: 5000 }
  })
}

function show(name: string, chars: Array<string>, plan: typeset.ParagraphPlan) {
  io.writeLine("{name} {plan.totalCost}")
  for line in plan.lines {
    io.writeLine("  {chars.slice(line.start, line.trimmedEndExclusive).join("")} {line.trimmedTextWidth}")
  }
}

let chars = "줄을 하나씩 채우면 마지막 줄이 짧게 남습니다".scalars()
match plan(chars, 100000) {
  case Ok(result) => {
    show("greedy", chars, result.greedy)
    show("optimized", chars, result.optimized)
  };
  case Err(error) => io.writeLine("{error}")
}

Output

greedy 685996370
  줄을 하나씩 채우면 85000
  마지막 줄이 짧게 75000
  남습니다 40000
optimized 115677384
  줄을 하나씩 52500
  채우면 마지막 줄이 85000
  짧게 남습니다 62500

The greedy plan fills the first two lines and leaves the single word 남습니다 on the last line. A last line shorter than 52% of the line width costs much more, so the optimized plan shortens the earlier lines and evens out all three. The numbers are each plan's total cost and each line's width without trailing spaces.

breakParagraph returns both plans. Pick the plan to use from optimized and greedy in the ParagraphResult, and pick how cost is computed with Paragraph.costModel.

Cost model	Input	Use it for
`CostModel.FixedPoint`	Boxes alone are enough. The grid must be `MilliPoint`.	Korean, Chinese, and Japanese paragraphs with no glue between characters, and paragraphs whose word spacing is rated through `latinSpacing`
`CostModel.KnuthPlass`	A line whose natural width plus the chosen penalty's `width` equals the target needs no glue. A shorter line needs glue that can stretch, and a longer line needs glue that can shrink.	Paragraphs justified by stretching and shrinking the spaces between words

Knuth-Plass is the line breaking algorithm by Donald Knuth and Michael Plass. It rates each line by how far its glue must stretch or shrink, called badness, and picks the combination with the lowest cost for the whole paragraph. The KnuthPlass model compares every earlier break position as a candidate. The FixedPoint model computes cost in whole millipoints. Its optimized plan never ends three lines in a row at soft hyphens. Both models pick the candidate added first when costs tie.

If no greedy plan exists, breakParagraph returns NoGreedyPlan even when an optimized plan exists. This happens with KnuthPlass when toleranceR is low. Raise toleranceR or widen the glue's stretch.

Cost

A plan's cost is an integer, and a lower cost means a better plan. In the FixedPoint model a line costs the value lineBadnessMpt computes, plus n as given when the chosen penalty is Allowed(n). Call lineBadnessMpt to compute the cost of a single line.

TOPAZ

import std.io
import std.typeset

let target = 100000
let space = typeset.BreakKind.LatinSpace
io.writeLine("fit    {typeset.lineBadnessMpt(100000, target, 10, 2, space, false, 0, 0)}")
io.writeLine("short  {typeset.lineBadnessMpt(90000, target, 9, 2, space, false, 0, 0)}")
io.writeLine("hyphen {typeset.lineBadnessMpt(100000, target, 10, 2, typeset.BreakKind.LatinHyphen, false, 0, 0)}")
io.writeLine("last   {typeset.lineBadnessMpt(40000, target, 4, 0, typeset.BreakKind.ForcedEnd, true, 0, 0)}")
io.writeLine("zero   {typeset.lineBadnessMpt(100000, 0, 10, 2, space, false, 0, 0)}")

Output

fit    Ok(10000000)
short  Ok(12770080)
hyphen Ok(75000000)
last   Ok(654000000)
zero   Err(TypesetError.InvalidInput(target must be positive; counts must be nonnegative))

The table below lists the costs of the FixedPoint model and lineBadnessMpt, and B is 1,000,000. A line that fits exactly costs 10B, and a line that ends at a hyphen adds 65B for 75B. A last line that fills only 40% of the width adds 500B. A target width of 0 returns InvalidInput.

Condition	Added cost
Every line	10B
Gap between line width and target width	Grows with the square of the gap ratio. More word spaces make it smaller.
Line wider than the target	20,000B or more
Line ending at a hyphen or soft hyphen	65B
Line broken at a `HardFallback` position	50,000B
Line with two characters or fewer	275B
Last line	The gap cost drops to two fifths. Adds 500B when shorter than 52% of the target.
`justificationB` argument	Added as given

The KnuthPlass model computes an adjustment ratio r for each line. The leftover width is the target minus the line's natural width and the chosen penalty's width. r is the leftover width divided by the total stretch when it is 0 or more, and by the total shrink when it is negative. A line with r below −1 or above toleranceR / 100,000,000 is dropped. When the total stretch or shrink to divide by is 0, only a line with exactly no leftover width is accepted. Badness is 100 × |r|³ rounded, capped at 10,000. A line costs (linePenalty + badness)² plus the following.

For a penalty Allowed(n), add n² when n is 0 or more and subtract n² when n is negative.
Add fitnessDemerit when two neighboring lines differ by two or more fitness classes. The four classes are r below −0.5, up to 0.5, up to 1, and above 1.
Add flaggedDemerit when two lines in a row end at flagged penalties.
Add finalFlaggedDemerit when the previous line ended at a flagged penalty and this line is the last line of the paragraph.

Integer overflow does not return a TypesetError. It stops the program with the runtime error TPZ4004.

Justify spacing

Justification spreads each line's leftover width over its spaces so that the line ends flush. In the KnuthPlass model, add spaces as Glue. A glue item has a natural width, a stretch it can grow by, and a shrink it can lose. Assuming a 10pt font, the next example uses a space of 3,333, a stretch of 1,667, and a shrink of 1,111 millipoints.

TOPAZ

import std.io
import std.typeset

let space = 3333
let stretch = 1667
let shrink = 1111

function plan(chars: Array<string>, width: int) -> Result<typeset.ParagraphResult, typeset.TypesetError> {
  let mut items: Array<typeset.Item> = []
  for i in 0..<chars.length {
    if chars[i] == " " {
      items.push(typeset.Item.GlueItem(typeset.Glue { unitIndex: i, text: " ", width: space, trackingAfter: 0, stretch: stretch, shrink: shrink }))
      items.push(typeset.Item.PenaltyItem(typeset.Penalty { unitEnd: i + 1, width: 0, rule: typeset.BreakRule.Allowed(0), kind: typeset.BreakKind.LatinSpace, flagged: false }))
    } else {
      items.push(typeset.Item.BoxItem(typeset.Box { unitIndex: i, text: chars[i], width: 5000, trackingAfter: 0 }))
    }
  }
  let end = chars.length
  items.push(typeset.Item.GlueItem(typeset.Glue { unitIndex: end, text: "", width: 0, trackingAfter: 0, stretch: 100000000, shrink: 0 }))
  items.push(typeset.Item.PenaltyItem(typeset.Penalty { unitEnd: end + 1, width: 0, rule: typeset.BreakRule.Forced, kind: typeset.BreakKind.ForcedEnd, flagged: false }))

  typeset.breakParagraph(typeset.Paragraph {
    grid: typeset.DistanceGrid.MilliPoint,
    items: items,
    geometry: typeset.LineGeometry { bodyTarget: width, firstTarget: width, dropCapSpan: 0, dropCapInset: 0 },
    justification: typeset.JustificationPolicy {
      mode: typeset.JustificationMode.Unjustified, minGapCount: 0, minLineWidthRatioM: 0,
      minFinalLineWidthRatioM: None, maxAdjustment: 0, maxGapAdjustment: 0, maxGapRatioR: 0,
      latinSpacing: None, cjkStretchRatio: 0.0
    },
    costModel: typeset.CostModel.KnuthPlass,
    knuthPlass: typeset.KnuthPlassPolicy { toleranceR: 200000000, linePenalty: 10, fitnessDemerit: 10000, flaggedDemerit: 10000, finalFlaggedDemerit: 5000 }
  })
}

let chars = "Typesetting splits a paragraph into lines and then spreads the spaces in each line evenly.".scalars()
match plan(chars, 150000) {
  case Ok(result) => {
    for line in result.optimized.lines {
      let r = line.adjustmentRatioR
      let gap = if r >= 0 { space + r * stretch / 100000000 } else { space + r * shrink / 100000000 }
      io.writeLine("{chars.slice(line.start, line.trimmedEndExclusive).join("")} | {r} {gap}")
    }
  };
  case Err(error) => io.writeLine("{error}")
}

Output

Typesetting splits a paragraph | 100000000 5000
into lines and then spreads the | 40011998 4000
spaces in each line evenly. | 21667 3333

adjustmentRatioR is the adjustment ratio r multiplied by 100,000,000. When r is 0 or more a space is width + r × stretch wide, and when r is negative it is width + r × shrink. The first line has r = 1, so its spaces grow from 3,333 to 5,000. The second line has r of about 0.40, so its spaces are 4,000.

The last line ends with glue that has a very large stretch, followed by a Forced penalty, so it stays flush left. A Forced penalty keeps the glue right before it on the line, which brings the last line's r close to 0. At an ordinary break the glue at the end of the line is dropped, and glue at the start of the next line is skipped.

The FixedPoint model rates justification through JustificationPolicy instead of glue. With latinSpacing, it assumes word spaces can shrink to minWordSpaceRatio times and stretch to maxWordSpaceRatio times their natural width. Letters can spread apart by up to maxLetterSpacingEm. The allowed ranges are 0.8 to 1.0, 1.0 to 1.33, and 0 to 0.015, and values outside them return InvalidPolicy. When the mode is not Unjustified and latinSpacing is present, the optimized plan first looks for a plan that fills every line except the last within the range the justification policy allows. The last line is never checked. If such a plan exists it is used and optimized.boundedFillFeasible is Some(true). Otherwise the search runs again without the range condition and the field is Some(false). In any other case it is None, and the greedy plan's boundedFillFeasible is always None.

Locales

Choose the kinsoku rules for a language with BreakPolicy.rules. The three Unicode rule sets keep the non-tailorable decisions of UAX #14 and add the language's kinsoku on top.

Value	Rules
`Unicode16Ko`	KLREQ, the W3C Requirements for Hangul Text Layout and Typography, 7.1.1 to 7.1.4. Keeps Korean words that fit in `Word` mode.
`Unicode16Zh`	CLREQ, the W3C Requirements for Chinese Text Layout, with the basic kinsoku of 6.1.1
`Unicode16Ja`	The strict kinsoku of JLREQ 3.1.7 to 3.1.10, symbols before and after numbers, and digit grouping spaces inside numbers
`AtlasDefault`	The rule set ATLAS has used. It uses fixed punctuation lists instead of Unicode tables.

The next example breaks the same text with Unicode16Ko, Unicode16Zh, and Unicode16Ja, in that order.

TOPAZ

import std.io
import std.typeset

function marked(text: string, rules: typeset.KinsokuRules, mode: typeset.BreakMode) -> string {
  let chars = text.scalars()
  let clusters = chars.map(ch => typeset.BreakCluster { text: ch, advance: 10000, trackingAfter: 0 })
  let policy = typeset.BreakPolicy {
    mode: mode, rules: rules, forbidBefore: [], forbidAfter: [],
    boundaryAllowed: chars.map(ch => true), generatedHyphenEnds: [],
    protectFittingLatin: false, preserveShortWords: false,
    narrowestWordTarget: 100000, bodyWordTarget: 100000
  }
  match typeset.classifyBreaks(clusters, policy) {
    case Ok(points) => {
      let ends = points.map(p => p.unitEnd)
      let mut out = ""
      for i in 0..<chars.length {
        out = "{out}{chars[i]}"
        if i + 1 < chars.length && ends.indexOf(i + 1) != None { out = "{out}|" }
      }
      out
    };
    case Err(error) => "{error}"
  }
}

let rules = [typeset.KinsokuRules.Unicode16Ko, typeset.KinsokuRules.Unicode16Zh, typeset.KinsokuRules.Unicode16Ja]
for text in ["1 000", "#一", "1㎏"] {
  io.writeLine(rules.map(r => marked(text, r, typeset.BreakMode.Character)).join("   "))
}

Output

1 |000   1 |000   1 000
#|一   #|一   #一
1|㎏   1|㎏   1㎏

Unicode16Ja keeps the prefix symbol # and the unit ㎏ attached to their numbers. It treats the space in 1 000 as a digit group separator and does not break there. Unicode16Ko and Unicode16Zh do not apply these Japanese rules.

Handle errors

classifyBreaks, breakParagraph, and lineBadnessMpt return Result<_, TypesetError>. They never change their inputs, and every result array is new.

Error	Returned when
`InvalidInput(message)`	The cluster list is empty, or `boundaryAllowed` has a different length. A cluster contains a mandatory break, or `boundaryAllowed` blocks one. `unitIndex` does not run from 0 without gaps. Glue has a negative stretch or shrink. A penalty points outside the paragraph. The target width passed to `lineBadnessMpt` is 0 or less.
`InvalidPolicy(message)`	A `LineGeometry` target width is 0 or less, or `dropCapSpan` is outside 0 to 3. `FixedPoint` gets a grid other than `MilliPoint`. A justification, spacing, or Knuth-Plass value is out of range. `forbidBefore` or `forbidAfter` holds a value that is not a Unicode scalar.
`NoBreakOpportunities`	No break opportunity can be chosen.
`NoGreedyPlan`	The greedy plan cannot be built through to the end of the paragraph.
`NoPlan`	The optimized plan finds no way to break lines through to the end of the paragraph. The optimized plan of `FixedPoint` drops lines wider than `postValidationWidthLimit`, so too few break positions lead to this error.

Public API

After import std.typeset, write names such as typeset.breakParagraph and typeset.BreakKind.ForcedEnd.

Functions

Function	Signature	Description
`breakParagraph`	`(input: Paragraph) -> Result<ParagraphResult, TypesetError>`	Plans the paragraph's optimized and greedy line breaks together.
`classifyBreaks`	`(clusters: Array<BreakCluster>, policy: BreakPolicy) -> Result<Array<BreakOpportunity>, TypesetError>`	Finds the break opportunity and its kind at each cluster boundary.
`lineBadnessMpt`	`(widthMpt: int, targetMpt: int, charCount: int, spaces: int, kind: BreakKind, isLast: bool, justificationB: int, hyphenAdvanceMpt: int) -> Result<int, TypesetError>`	Computes the line cost of the `FixedPoint` model.
`fromMilliPoints`	`(value: int) -> Length<Pt>`	Turns an integer number of millipoints into a length.
`fromScaledPoints`	`(value: int) -> Length<Pt>`	Uses an integer number of scaled points as a length unchanged.

Input types

Type	Fields	Description
`BreakCluster`	`text: string`, `advance: int`, `trackingAfter: int`	A cluster to classify and its width
`BreakPolicy`	`mode: BreakMode`, `rules: KinsokuRules`, `forbidBefore: Array<int>`, `forbidAfter: Array<int>`, `boundaryAllowed: Array<bool>`, `generatedHyphenEnds: Array<int>`, `protectFittingLatin: bool`, `preserveShortWords: bool`, `narrowestWordTarget: int`, `bodyWordTarget: int`	The classification policy. `forbidBefore` and `forbidAfter` list Unicode scalar values that a line must not break before or after. `boundaryAllowed` has one entry per cluster saying whether a line may break after it. `generatedHyphenEnds` lists boundaries inside words where a soft hyphen may go. `Word` mode drops `HardFallback` positions inside a word no wider than `bodyWordTarget`, and with `Unicode16Ko` rules it also drops the positions between Hangul inside a Korean word of that width. A word wider than `bodyWordTarget` gets `HardFallback` positions between its letters.
`Box`	`unitIndex: int`, `text: string`, `width: int`, `trackingAfter: int`	A unit of text that never stretches or shrinks
`Glue`	`unitIndex: int`, `text: string`, `width: int`, `trackingAfter: int`, `stretch: int`, `shrink: int`	A space unit that can stretch and shrink
`Penalty`	`unitEnd: int`, `width: int`, `rule: BreakRule`, `kind: BreakKind`, `flagged: bool`	A break position at the boundary after the first `unitEnd` units. It occupies no unit. `width` is added to the width of a line broken there. `KnuthPlass` always adds it, and `FixedPoint` adds it only when `kind` is `LatinSoftHyphen`. The added value is kept in `PlanLine.selectedPenaltyWidth`. `flagged` marks positions such as hyphens that should not repeat on consecutive lines, and only `KnuthPlass` counts it in the cost (`flaggedDemerit` and `finalFlaggedDemerit` in the cost section).
`Item`	`BoxItem(Box)`, `GlueItem(Glue)`, `PenaltyItem(Penalty)`	An item of a paragraph
`LineGeometry`	`bodyTarget: int`, `firstTarget: int`, `dropCapSpan: int`, `dropCapInset: int`	Target line widths. `firstTarget` applies to the first line and `bodyTarget` to the rest. The first `dropCapSpan` lines (0 to 3) subtract `dropCapInset` from their target.
`LatinSpacing`	`minWordSpaceRatio: float`, `maxWordSpaceRatio: float`, `maxLetterSpacingEm: float`, `fontSizePt: float`	Allowed ranges for Latin word spacing and letter spacing
`JustificationPolicy`	`mode: JustificationMode`, `minGapCount: int`, `minLineWidthRatioM: int`, `minFinalLineWidthRatioM: Option<int>`, `maxAdjustment: int`, `maxGapAdjustment: int`, `maxGapRatioR: int`, `latinSpacing: Option<LatinSpacing>`, `cjkStretchRatio: float`	The justification policy of the `FixedPoint` model. The `KnuthPlass` model only validates it.
`KnuthPlassPolicy`	`toleranceR: int`, `linePenalty: int`, `fitnessDemerit: int`, `flaggedDemerit: int`, `finalFlaggedDemerit: int`	The tolerance (× 100,000,000) and cost terms of the `KnuthPlass` model
`Paragraph`	`grid: DistanceGrid`, `items: Array<Item>`, `geometry: LineGeometry`, `justification: JustificationPolicy`, `costModel: CostModel`, `knuthPlass: KnuthPlassPolicy`	The paragraph passed to `breakParagraph`

Result types

Type	Fields	Description
`BreakOpportunity`	`unitEnd: int`, `kind: BreakKind`	A break opportunity at the boundary after the first `unitEnd` clusters
`PlanLine`	`start: int`, `endExclusive: int`, `trimmedEndExclusive: int`, `displayTextScalarLength: int`, `kind: BreakKind`, `target: int`, `rawTextWidth: int`, `trimmedTextWidth: int`, `costTextWidth: int`, `selectedPenaltyWidth: int`, `postValidationWidthLimit: Option<int>`, `cost: int`, `adjustmentRatioR: int`	One line. `costTextWidth` is the width used for its cost. `postValidationWidthLimit` is the width check limit of `FixedPoint`. It is `target` + 1 for the last line, and for other lines `target` plus the shrink computed from `latinSpacing` (0 when there is none) plus 1. The optimized plan drops a line whose width without trailing spaces exceeds this value, counting the penalty `width` at a soft hyphen. It never drops a line of a single unit. The greedy plan records the value without checking it. So a single unit line and a greedy line can be wider than this value. It is `None` under `KnuthPlass`. `adjustmentRatioR` is 0 under `FixedPoint`.
`ParagraphPlan`	`lines: Array<PlanLine>`, `totalCost: int`, `boundedFillFeasible: Option<bool>`, `grid: DistanceGrid`	One line plan and its total cost. `boundedFillFeasible` is the bounded fill result described in the spacing section. Only the optimized plan records it, and it is `None` in the greedy plan.
`ParagraphResult`	`optimized: ParagraphPlan`, `greedy: ParagraphPlan`, `breakOpportunityCount: int`	The optimized plan, the greedy plan, and the number of selectable penalties

Enums

Type	Values	Description
`BreakKind`	`KoreanCluster`, `LatinSpace`, `LatinHyphen`, `LatinSoftHyphen`, `ClosingPunctuation`, `HardFallback`, `ForcedEnd`	The kind of a break opportunity
`BreakMode`	`Word`, `Character`	`Word` applies the word protection described under `BreakPolicy`. `Character` gives every position UAX #14 and the kinsoku rules allow, without that protection. Inside a Latin word such as `Hello`, where UAX #14 gives no break, `Character` does not break either.
`BreakRule`	`Allowed(int)`, `Forced`, `Forbidden`	How a penalty behaves. `Forced` always breaks, and `Forbidden` never breaks. The n in `Allowed(n)` is the cost of breaking there. `FixedPoint` adds n as given, and `KnuthPlass` adds n² when n is 0 or more and subtracts n² when n is negative.
`CostModel`	`FixedPoint`, `KnuthPlass`	How line cost is computed
`DistanceGrid`	`ScaledPoint`, `MilliPoint`	The unit of width integers
`JustificationMode`	`Unjustified`, `LatinWordSpace`, `CjkInterCharacter`	No justification, justification through word spaces, or justification between CJK characters
`KinsokuRules`	`AtlasDefault`, `Unicode16Ko`, `Unicode16Zh`, `Unicode16Ja`	The line breaking rule set
`TypesetError`	`InvalidInput(string)`, `InvalidPolicy(string)`, `NoBreakOpportunities`, `NoGreedyPlan`, `NoPlan`	Errors from the typesetting functions