serhii.net

EDIT: this is becoming a more generic thingy for everything I’d ever need to refer to when writing a paper, TODO clean this at some point.

Resources – DREAM Lab links to https://dream.cs.umass.edu/wp-content/uploads/2020/04/Tips-and-Best-Practices.pdf. Until I set up a system to save PDF info, I’ll paste it as screenshots here:

ChatGPT summarized the relevant pages of the PDF file thus, but didn’t do it well, mostly rewriting myself:

General Formatting

• Smart Quotes: Use `x’ and ``x’’ for single and double quotes respectively.
• Large Numbers: Write large numbers as 54{,}000.
• Word Breaking: Use - to suggest hyphenation points in long words, including ones that already have hyphens inside them. Such words are likely to go over margins etc.: multi-discipli\-nary
  • EDIT: If I don’t want a word to get broken, use mbox: \caption{Worldwide distribution of \mbox{plantNaturalist-}500k}¹
  • EDIT2: the extdash package is even better, same link
  • EDIT3: If you want to allow linebreaks somewhere but without a dash and only if needed, it’s long\allowbreak word²
• Overflow Lines: \begin{sloppypar}... for paragraphs where latex goes over the margin.
• Float Positioning: Place figures and tables at the top of the page for readability: \begin{figure}[t]
• Center Align Floats: Use \centering for aligning tables and figures.

Specific Elements

• Non-Breaking Space: Use the tilde (~) to keep words together, including always
  • before citations: sth~\cite{whatever}
  • after numbers in text
  • before figure/section references
• Emphasize vs. Bold: Prefer \emph over bold or \textit.

Equations and Macros

• Macros: Create macros for things that will be changed later, such as DATASET NAMES. \newcommand{\system}{SQuID\xspace}
  • \xspace here adds as space unless end of sentence. Package \usepackage{xspace}

Spacing and Sizing

• Paragraph Spacing: Use \smallskip, \medskip, and \bigskip, instead of \vspace
• Line and Text Width: Use fractions of \linewidth or \textwidth.
• Resizing Tables/Figures: Use \resizebox with appropriate dimensions.

Compression hacks

• (see pictures)
• Make spaces smaller around lists:

\begin{itemize}
  \setlength{\itemsep}{0pt}
  \setlength{\parskip}{0pt}

Paper writing hacks

Even more

Non-breaking spaces

best practices - When should I use non-breaking space? - TeX - LaTeX Stack Exchange lists ALL the places where Knuth wanted people to put nonbreaking spaces, incl:

• 1)~one 2)~two
• Donald~E. Knuth
• 1,~2
• Chapter~12

Less obvious and not from him:

• I~am

Also:

• before all cites, refs
• … and inlined equations

Citations

• ChatGPT says that citations should come before footnotes to prioritize the scholarly source over unimportant info.
  • So this [32] ³ and not this³ [32].
• Citations go before the last period of the sentence. EDIT: NOT PARENTHESES, THEY SHOULD BE WITHIN PARENTHESES.⁴ DAMN

Other bits

• I sometimes write and around ~50% forgetting that ~ is a nbsp — hard to catch when reading the text.
• As when writing code I like to add some assert False (or a failing test) so that I know where I stopped the last time, \latexstopcompiling here is a neat way to make sure I REALLY finis ha certain line I started but not finished.

Rounding.

Previously: 211018-1510 Python rounding behaviour with TL;DR that python uses banker’s rounding, with .5th rounding towards the even number.

• Rounding - Wikipedia

• Nearest Integer Function – from Wolfram MathWorld

• Floor/ceil have their usual latex notation as \rceil, \rfloor (see LaTeX/Mathematics - Wikibooks, open books for an open world at ‘delimiters’)

• “Normal” rounding (towards nearest integer) has no standard notation: ceiling and floor functions - What is the mathematical notation for rounding a given number to the nearest integer? - Mathematics Stack Exchange

  • $\lfloor3.54\rceil$ is one notation people mention
  • Some suggest $[3.54]$ but most hate this because overused
  • $nint(3.45)$ (’nearest integer function’) works
  • let XXX denote the standard rounding function

• Bankers’ rounding (that python and everyone else use for tie-breaking for normal rounding and .5) has no standard notation as well

  • Let $\lfloor x \rceil$ denote "round half to even" rounding (a.k.a. "Banker's rounding"), consistent with Python's built-in round() and NumPy's np.round() functions.

Require/ensure

Require/Ensure is basically Input/Output and can be renamed thus¹:

\floatname{algorithm}{Procedure}
\renewcommand{\algorithmicrequire}{\textbf{Input:}}
\renewcommand{\algorithmicensure}{\textbf{Output:}}

Sample algpseudocode algorithm float

\usepackage{algorithm}
\usepackage{algpseudocode}

% ...

\begin{algorithm}
\caption{Drop Rare Species per Country}
\label{alg:drop}
\begin{algorithmic}
\Require $D_0$: initial set of occurrences
\Ensure $D_1$: Set of occurrences after filtering rare species
\State $D_1 \gets$ \emptyset
\For{each $c$ in Countries}
    \For{each $s$ in Species}
        \If {$|O_{c,s} \in D_0| \geq 10$} % if observations of species in country in D_0 have more than 10 entries; || is set cardinality
            \State{$D_1 \gets D_1 \cup O_{c,s}$} 
        \EndIf
    \EndFor
\EndFor
\end{algorithmic}
\end{algorithm}

List of all map providers, not all included in geopandas and some paid, nevertheless really neat: https://xyzservices.readthedocs.io/en/stable/gallery.html

For the 231024-1704 Master thesis task CBT task of my 230928-1745 Masterarbeit draft, I’d like to create an ontology I can use to “seed” LMs to generate ungoogleable stories.

And it’s gonna be fascinating.

I don’t know what’s the difference between knowledge graph, ontology etc. at this point.

Basics

I want it to be highly abstract - I don’t care if it’s a forest, if it’s Cinderella etc., I want the relationships.

Let’s try. Cinderella is basically “Rags to riches”, so:

• Character roles
  • Protagonist
    • Underprivileged protagonist (Cinderella)
  • Benefactor
  • …
• Key plot points
  • Hardship
  • Opportunity
  • Transformation
  • Achievement
• Thematic elements
  • Transformation
  • Reward?..

…

Or GPT3’s ideas from before:

"Entities": {
	"Thief": {"Characteristics": ["Cunning", "Resourceful"], "Role": "Protagonist"},
	"Fish": {"Characteristics": ["Valuable", "Symbolic"], "Role": "Object"},
	"Owner": {"Characteristics": ["Victimized", "Unaware"], "Role": "Antagonist"}
},
"Goals": {
	"Thief": "Steal Fish",
	"Owner": "Protect Property"
},
"Challenges": {
	"Thief": "Avoid Detection",
	"Owner": "Secure Property"
},
"Interactions": {
	("Thief", "Fish"): "Theft",
	("Thief", "Owner"): "Avoidance",
	("Owner", "Fish"): "Ownership"
},
"Outcomes": {
	"Immediate": "Successful Theft",
	"Long-term": "Loss of Trust"
},
"Moral Lessons": {
	"Actions Have Consequences",
	"Importance of Trust",
	"Greed Leads to Loss"
}

• ENTITIES have a:
  • ROLE
    • Protagonist
  • CHARACTERISTICS
    • Poor

Awesome chats with ChatGPT

Here’s it generating an ontology based on the above graph: https://chat.openai.com/share/92ed18ce-88f9-4262-9dd9-f06a07d06acc

And more in UKR: https://chat.openai.com/share/846a5e85-353e-4bb5-adbe-6da7825c51ed

Tale ontology v0.0.1

In bold bits I’m not sure of. In decreasing order of abstraction, with the first two being the most generic ones and the latter ones more fitting for concrete stories.

• KEY PLOT POINTS:
  • Hardship
  • Disruption
  • Opportunity
  • Transformation
  • Achievement
• CHARACTER ROLES:
  • Description: most abstract roles for the characters
  • Values:
    • Protagonist
    • Antagonist
    • Object
    • Benefactor
    • Goal object
    • Ally
    • etc.
• ENTITIES
  • Description:
    • can be characters, important objects, whatever
  • Attributes:
    • Characteristics:
      • … basically adjectives.
      • e.g. Resourceful, motivated, hungry
    • Role: CHARACTER ROLE
• GOALS
  • Description: what drives the character/entity
  • Attributes:
    • Entity: ENTITY
    • Goal: main goal of entity in this context
  • Example:
    • Thief: steal fish
    • Owner: protect property
• INTERACTIONS
  • Plaintext description of what’s up between pairs of ENTITIES
  • Example: (’thief’,‘fish’): theft
• OUTCOMES
  • Attributes:
    • SHORT-TERM: plaintext description
    • LONG-TERM: plaintext description
  • Example: “loss of trust”
  • Optional
• MORAL LESSONS:
  • Description: short plaintext description
  • Optional
  • examples:
    • “Actions Have Consequences”, “Importance of Trust”, “Greed Leads to Loss”

Remaining issues:

• Can I generalize to remove “fish” and stuff? Maybe call it object?
• Two versions of the ontology with different levels of abstractions?

Test 1:

Here’s ChatGPT applying that to Shrek: https://chat.openai.com/share/d96d4be6-d42f-4096-a18f-03f786b802c6

Modifying its answers:

“Using this ontology for abstract fairy tale description, please create a generalized graph structure for THE FIRST HARRY POTTER MOVIE. Focus on the overarching themes and character roles without specific names or unique settings. The graph should include key plot points, character roles, entities, goals, interactions, outcomes, and moral lessons, all described in a manner that is broadly applicable to similar stories.”

Revised Ontology for Abstract Fairy Tale Description

Character Roles

• Protagonist: Central character facing a challenge.
• Antagonist: Character or force opposing the protagonist.
• Goal Object: The objective or person the protagonist seeks.
• Ally: A supportive character aiding the protagonist.

Entities

• Description: Characters or significant elements in the story.
• Attributes:
  • Characteristics: Abstract adjectives describing the entity.
  • Role: One of the defined character roles.

Goals

• Description: Motivations driving the entities.
• Attributes:
  • Entity: Reference to an entity.
  • Objective: Abstract goal of the entity.

Interactions

• Description: General interactions between pairs of entities.
• Format: (‘Entity1’, ‘Entity2’): Type of interaction.

Outcomes

• Short-term: Immediate result of the story’s climax.
• Long-term: Lasting changes or lessons from the story.

Moral Lessons

• Description: Universal themes or morals derived from the story.

Attempt 2

<

Tools

• Tutorial: Build a Knowledge Graph using NLP and Ontologies - Developer Guides

Context:

• 230928-1745 Masterarbeit draft,
• 231225-2240 Glosses markdown magic, 231226-1702 Ideas for annotating glosses in my Masterarbeit

> pandoc 230928-1745\ Masterarbeit\ draft.md -o master_thesis.pdf
# unicode magic
Try running pandoc with --pdf-engine=xelatex.
# thank you

> pandoc 230928-1745\ Masterarbeit\ draft.md -o master_thesis.pdf --pdf-engine=xelatex
# a volley of...
[WARNING] Missing character: There is no о (U+043E) in font [lmroman10-italic]:mapping=tex-text;!

• Ugly:
  • 
  • 

Makefile magic etc

Exporting Hugo to PDF | akos.ma looks nice.

build/pdf/%.pdf: content/posts/%/index.md
	$(PANDOC) --write=pdf --pdf-engine=xelatex \
		--variable=papersize:a4 --variable=links-as-notes \
		--variable=mainfont:DejaVuSans \
		--variable=monofont:DejaVuSansMono \
		--resource-path=$$(dirname $<) --out=$@ $< 2> /dev/null

Let’s try:

pandoc 230928-1745\ Masterarbeit\ draft.md -o master_thesis.pdf --pdf-engine=xelatex  --variable=links-as-notes \
--variable=mainfont:DejaVuSans \
--variable=monofont:DejaVuSansMono

Better but not much; HTML is not parsed, lists count as lists only after a newline it seems.

Pandoc’s Markdown requires a newline after a paragraph for a list to render · Issue #6590 · jgm/pandoc

pandoc 230928-1745\ Masterarbeit\ draft.md -o master_thesis.pdf --pdf-engine=xelatex  --variable=links-as-notes \
--variable=mainfont:DejaVuSans \
--variable=monofont:DejaVuSansMono \
--from=markdown+lists_without_preceding_blankline

Better, but quotes unsolved:

Markdown blockquote shouldn’t require a leading blank line · Issue #7069 · jgm/pandoc

pandoc 230928-1745\ Masterarbeit\ draft.md -o master_thesis.pdf --pdf-engine=xelatex  --variable=links-as-notes \
--variable=mainfont:DejaVuSans \
--variable=monofont:DejaVuSansMono \
--from=markdown+lists_without_preceding_blankline
#+blank_before_blockquote

ACTUALLY, - f gfm (github-flavour) solves basically everything. commonmark doesn’t parse latex, commonmark_x (‘with many md extensions’) on first sight is similar to gfm.

I think HTML is the last one.

Raw HTML says it’s only for strict:

--from=markdown_strict+markdown_in_html_blocks

msword - Pandoc / Latex / Markdown - TeX - LaTeX Stack Exchange suggest md to tex and tex to pdf, interesting approach.

6.11 Write raw LaTeX code | R Markdown Cookbook says complex latex code may be too complex for markdown.

This means this except w/o backslashes:

\```{=latex}
$\underset{\text{NOUN-NOM}}{\overset{\text{man}}{\text{чоловік-}\varnothing}}$ $\underset{\text{PST}}{\overset{\text{saw}}{\text{побачив}}}$ $\underset{\text{NOUN-ACC}}{\overset{\text{dog}}{\text{собак-у}}}$.
\```

Then commonmark_x can handle that.

EDIT: --standalone!

More on HTML sub/sup to PDF

I don’t need HTML, I need <sub>.

• pandoc md has a syntax for this: Pandoc - Pandoc User’s Guide

  • …but I’m not using pandoc md :(

• Options

  • Can I replace my tags w/ that with yet another filter?
  • Ignore that obsidian/hugo can’t parse them and use pandoc syntax?.. and do --from=markdown+lists_without_preceding_blankline+blank_before_blockquote? :(
  • export to HTML w/ mathjax and from it PDF?
  • just use latex syntax everywhere? :(

ChatGPT tried to create a filter but nothing works, I’ll leave it for later: https://chat.openai.com/share/c94fffbe-1e90-4bc0-9e97-6027eeab281a

• I’m not alone:
  • Pandoc style superscript and subscript - Feature requests - Obsidian Forum

HTML

This produces the best HTML documents:

> pandoc 230928-1745\ Masterarbeit\ draft.md -o master_thesis.html \
--from=gfm --mathjax --standalone

NB If I add CSS, it should be an absolute path:

Later

Convert Markdown to PDF

• 6.2 Pandoc options for LaTeX output | R Markdown Cookbook
  • This has config options for latex, incl. fonts Pandoc - Pandoc User’s Guide
• 6.4 Include additional LaTeX packages | R Markdown Cookbook what a cool place

Callouts

It’d be cool to wrap examples in the same environment!

https://forum.obsidian.md/t/rendering-callouts-similarly-in-pandoc/40020:

-- https://forum.obsidian.md/t/rendering-callouts-similarly-in-pandoc/40020/6
--
local stringify = (require "pandoc.utils").stringify

function BlockQuote (el)
    start = el.content[1]
    if (start.t == "Para" and start.content[1].t == "Str" and
        start.content[1].text:match("^%[!%w+%][-+]?$")) then
        _, _, ctype = start.content[1].text:find("%[!(%w+)%]")
        el.content:remove(1)
        start.content:remove(1)
        div = pandoc.Div(el.content, {class = "callout"})
        div.attributes["data-callout"] = ctype:lower()
        div.attributes["title"] = stringify(start.content):gsub("^ ", "")
        return div
    else
        return el
    end
end

Makes:

> [!NOTE]- callout Title
>
> callout content

into

::: {.callout data-callout="note" title="callout Title"}
callout content
:::

.callout {
    color: red; /* Set text color to red */
    border: 1px solid red; /* Optional: add a red border */
    padding: 10px; /* Optional: add some padding */
    /* Add any other styling as needed */
}

Then this makes it pretty HTML:

pandoc callout.md -L luas/obsidian-callouts.lua -t markdown -s | pandoc --standalone -o some_test.html --css luas/callout-style.css

<div class="callout" data-callout="note" title="callout Title">
<p>callout content</p>
</div>

For PDF: .. it’s more complex, will need such a header file etc. later on. TODO

\usepackage{xcolor} % Required for color definition
\newenvironment{callout}{
  \color{red} % Sets the text color to red within the environment
  % Add any other formatting commands here
}{}

Unrelated

Footnotes

• What if I put the footnotes in the text margins and do a HTML-first thesis?
• margin notes!
  • LaTeX/Footnotes and Margin Notes - Wikibooks, open books for an open world:
  • Margin notes like footnotes?
  • discussion with css: Writing a side note on R markdown html #2 - R Markdown - Posit Community

Tufte CSS with pandoc

• A Tufte Handout Example
• tufte-css/tufte.css at gh-pages · edwardtufte/tufte-css
  • Which I can add as --css /abs/tufte.css!
• even better: jez/tufte-pandoc-css: Starter files for using Pandoc Markdown with Tufte CSS
  • hard install process though
  • very hard
• It uses this: jez/pandoc-sidenote: Convert Pandoc Markdown-style footnotes into sidenotes
  • Copied executables to /home/sh/.local/bin/: aha so that’s where you put your filters, inside $PATH

Damn! Just had to replace index.md with my thesis, then make all and it just …worked. Wow.

Apparently to make it not a sidenote I just have to add - to the footnote itself. Would be trivial to replace with an @ etc., then I get my inital plan - citations as citations and footnotes with my remarks as sidenotes.

I can add --from gfm --mathjax to the makefile command and it works with all my other requirements!

pandoc \
	--katex \
	--section-divs \
	--from gfm \
	--mathjax \
	--filter pandoc-sidenote \
	--to html5+smart \
	--template=tufte \
	--css tufte.css --css pandoc.css --css pandoc-solarized.css --css tufte-extra.css \
	--output docs/tufte-md/index.html \
	docs/tufte-md/index.md

I wonder if I can modify it to create latex-style sidenotes, it should be very easy: pandoc-sidenote/src/Text/Pandoc/SideNote.hs at master · jez/pandoc-sidenote

Numering references etc

• lierdakil/pandoc-crossref: Pandoc filter for cross-references
  • pandoc-crossref(1) | pandoc-crossref

![Caption](file.ext){#fig:label}
$$ math $$ {#eq:label}
 Section {#sec:section}

TODO figure out, and latex as well.

Citations

TODO

Current best:

eng: the man^NOM.SG saw the dog^NOM.SG
ukr: чоловік^man-NOM.SG побачив^saw-PST собакy^dog-ACC.SG

Ideas

I’d love to integrate the usual UD feats bits but they take a lot of space, and it’s either latex magic or one word per line.

• ukr: чоловік^{(man): Case=Nom|Number=Sing} побачив^(saw) собакy^{(dog): Case=Acc|Number=Sing}

• $чоловік^{man}_{Case=Nom|Number=Sing}$

• ${\underset{man}{чоловік}}^{Case=Nom|Number=Sing}$

• $\underset{Case=Nom|Number=Sing}{чоловік^{man}}$

• $\underset{NOM.SG}{чоловік^{man}}$

• ${\underset{man}{чоловік}}^{Case=Nom|Number=Sing}$

• ${\underset{man}{чоловік}}^{NOM.SG}$

• ${\underset{man}{чоловік}}^{NOM.SG}$ ${\underset{saw}{побачив}}$ ${\underset{dog}{собаку}}^{GEN.PL}$

я I _{Case=Nom|Number=Sing}
побачив saw
собаку saw  _{Animacy=Anim|Case=Acc|Gender=Masc|Number=Sing}

ukr: чоловік^man-NOM.SG побачив^saw-PST собакy^dog-GEN.PL

${\underset{man}{чоловік}}$ _{Case=Nom|Number=Sing} ${\underset{man}{чоловік}}$ _{Case=Nom|Number=Sing}

I think this is cool! But hell to write and parse:

$\underset{\text{NOUN.NOM}}{\overset{\text{man}}{\text{чоловік-}\varnothing}}$ $\underset{\text{PST}}{\overset{\text{saw}}{\text{побачив}}}$ $\underset{\text{NOUN-ACC}}{\overset{\text{dog}}{\text{собак-у}}}$.

$\underset{\text{NOUN.NOM}}{\overset{\text{man}}{\text{чоловік-}\varnothing}}$ $\underset{\text{PST}}{\overset{\text{saw}}{\text{побачив}}}$ $\underset{\text{NOUN-ACC}}{\overset{\text{dog}}{\text{собак-у}}}$.

Let’s play more with it:

$\underset{\text{Case=Nom|Number=Sing}}{\overset{\text{man }}{\text{чоловік}}}$ $\underset{\text{}}{\overset{\text{saw}}{\text{побачив}}}$ $\underset{\text{Case=Acc|Number=Sing}}{\overset{\text{dog}}{\text{собаку}}}$.

I can split it in diff lines: $\underset{\text{Case=Nom|Number=Sing}}{\overset{\text{man }}{\text{чоловік}}} \underset{\text{}}{\overset{\text{saw}}{\text{побачив}}} \underset{\text{Case=Acc|Number=Sing}}{\overset{\text{dog}}{\text{собаку}}}$.

$$\underset{\text{Case=Nom|Number=Sing}}{\overset{\text{man }}{\text{ЧОЛОВІК}}} \underset{\text{}}{\overset{\text{saw}}{\text{ПОБАЧИВ}}} \underset{\text{Case=Acc|Number=Sing}}{\overset{\text{dog}}{\text{СОБАКУ}}}$$

Splitting by morphemes

ukr: використовуватимуться ^{Aspect=Imp|Number=Plur|Person=3}

1 використовуватимуться використовуватися VERB _ Aspect=Imp|Mood=Ind|Number=Plur|Person=3|Tense=Fut|VerbForm=Fin 0 root _ SpaceAfter=No

ukr: використовуватимуть-ся^VERB-REFL

ukr: використовуватимуть^VERB -ся^REFL

$\underset{\text{NOM.SG}}{\overset{\text{man }}{\text{чоловік}}}$ $\underset{\text{PST}}{\overset{\text{saw}}{\text{побачив}}}$ $\underset{\text{SG-ACC}}{\overset{\text{dog}}{\text{собак-у}}}$.

This is one of the cooler ones, I’ll use it if I ever need to: Examples — graphviz 0.20.1 documentation

It’s also supported by HackMD! How to use MathJax & UML - HackMD

I’ll need something like overleaf for my markdown thesis.

5 Best Collaborative Online Markdown Editors - TechWiser

• HackMD
  • Markdown quick start guide - HackMD is nice!
  • The comments are part of the markdown code itself
    • but no way to highlight a certain phrase, just add a comment in a place of the text
  • you can share your doc to make it editable to not-logged-in users
  • Seems to hold my current thesis length quite well
  • I think I can stop looking