Diensttagebuch

Day 2339 (28 May 2025)

alacritty terminal

Kitty was really slow to start and this has been bugging me. Especially setting up a new system, using the default gnome-terminal, and seeing it appear instantly.

Kitty’s single instance mode (also -1) decreased start from 400ms to 300, still too much.

time alacritty -e bash -c exit
time gnome-terminal -e "bash -c exit"
time kitty --single-instance bash -c exit

New best friend: Alacritty

Saw Alacritty mentioned and it’s awesome. Has everything I wanted from or set up for kitty. Kitty is more configurable (I think), but I’m not missing anything at all so far.

Spawn instance in same directory

I used to have a separate command for that!

ArchWiki:

[keyboard]
bindings = [
   { key = "Return", mods = "Control|Shift", action = "SpawnNewInstance" }
]

Hints

Updated the default config to copy instead of launch, use better letters, and do file paths together with URIs:

[hints]
alphabet = "aoeusndh"

hints.enabled
action = "Copy"
# command = "xdg-open" # On Linux/BSD
hyperlinks = true
post_processing = true
persist = false
mouse.enabled = true
binding = { key = "N", mods = "Control|Shift" }

# adds file paths as well
regex = '(?:(?:ipfs:|ipns:|magnet:|mailto:|gemini://|gopher://|https?://|news:|git://|ssh:|ftp:|file:)[^\u0000-\u001F\u007F-\u009F<>"\s{}\-\^⟨⟩`\\]+|(?:(?:\.\.?/)+|/)[^\u0000-\u001F\u007F-\u009F<>"\s{}\-\^⟨⟩`\\]+)'
# regex = "(ipfs:|ipns:|magnet:|mailto:|gemini://|gopher://|https://|http://|news:|file:|git://|ssh:|ftp://)[^\u0000-\u001F\u007F-\u009F<>\"\\s{-}\\^⟨⟩`\\\\]+"

Vim mode

<C-S-Space> runs a vim-ish mode on the text, one can then copy etc. with all usual movements!

In kitty I had to do vim as scrollback pager etc. (old code below, prolly broken, didn’t use it because too complex)

For later

alacritty/alacritty-theme: Collection of Alacritty color schemes

Previously

Some old configs from kitty, for reference:

## Bindings

# https://sw.kovidgoyal.net/kitty/index.html#kittens
map kitty_mod>f1 launch --stdin-source=@screen_scrollback --stdin-add-formatting less +G -R

## Hints
# File paths
map kitty_mod+n>f kitten hints --type path --program @

# IPs (+ with ports)
map kitty_mod+n>i kitten hints --type regex --regex [0-9]+(?:\.[0-9]+){3} --program @
map kitty_mod+n>p kitten hints --type regex --regex [0-9]+(?:\.[0-9]+){3}:[0-9]+ --program @

# CLI Commands
# map kitty_mod+n>c kitten hints --type regex --regex "(\$|>)(.+)(?:\n|\s*$)?" --program @
# This version copies up to the vim mode indicator
# map kitty_mod+n>c kitten hints --type regex --regex "(\$|>)(.+?)(?:\n|\s+$|\s+(?:INS|VIS|REP|SEA))" --program @
# map kitty_mod+n>c kitten hints --type regex --regex "\$(.+)" --program @

# Linenum
map kitty_mod+n>l kitten hints --type line --program @

Scrollback/vim:

# https://sw.kovidgoyal.net/kitty/index.html#kittens
map kitty_mod>f1 launch --stdin-source=@screen_scrollback --stdin-add-formatting less +G -R
scrollback_pager vim - -c "w! /tmp/kitty_scrollback_sh" -c "term ++curwin cat /tmp/kitty_scrollback_sh"

Bash and zsh command substitution

After running a command (e.g. ag whatever, ^ag^rg re-runs it substituting the first part with the second — so rg whatever)
- Bash History: Correct / Repeat The Last Command With a Substitution - nixCraft
Fish doesn’t (want to) support this: Frequently asked questions — fish-shell 4.0.1 documentation

Ty BB for this

Day 2338 (27 May 2025)

taskfile go-task bits

zc
zc/it

Getting Started | Task

Day 2333 (22 May 2025)

jo cli util for creating json

Jan-Piet Mens :: A shell command to create JSON: jo

Day 2325 (14 May 2025)

neovim nvim, again

Lua-guide - Neovim docs
- tl;dr vim.opt.xxx = yy is the most conv. way to set all local global options.
- vim.wo etc. exist for easier access

vim.wo.relativenumber = true
vim.wo.cursorcolumn = true
vim.wo.cursorline = true
vim.wo.colorcolumn = "80"

-- :set tabstop=8 shiftwidth=4 softtabstop=4 expandtab shiftround
vim.g.tabstop = "8"
vim.g.softtabstop = "8"
vim.g.shiftwidth = "4"
vim.g.expandtab = true
vim.g.smarttab = true

Day 2316 (05 May 2025)

Evaluating RAG

General info

Roughly, it’s evaluating
- context — how relevant/correct are the retrieved chunks
- answer — how good are the generated claims
- (with interplay inbetween — e.g. whether the answer comes from the context, regardless of whether both are relevant to the query)
Or: generator metrics, retriever metrics, and overall metrics as used in RAGChecker (see RAGChecker picture later below).

Sources / libs

Evaluating - LlamaIndex
- Especially Usage Pattern (Response Evaluation) - LlamaIndex
  - Faithfullness: context(=source)+answer => whether the answer comes from the context available (=whether the answer was hallucinated)
  - Relevancy: query+context+answer => whether the answer AND CONTEXT were relevant for the specific query
- All LlamaIndex’s eval modules: Modules - LlamaIndex
  - CorrectnessEvaluator: compare query+answer to a reference answer: Correctness Evaluator - LlamaIndex w/ another LLM
  - Embedding Similarity Evaluator - LlamaIndex response+reference semantic similarity via embeddings

Deepeval’s metrics as given in the llamaindex docs:

from deepeval.integrations.llama_index import ( DeepEvalAnswerRelevancyEvaluator, DeepEvalFaithfulnessEvaluator, DeepEvalContextualRelevancyEvaluator, DeepEvalSummarizationEvaluator, DeepEvalBiasEvaluator, DeepEvalToxicityEvaluator, )

RAGChecker

RAGChecker: RAGChecker: A Fine-grained Framework For Diagnosing RAG
- LlamaIndex integration docs: RAGChecker: A Fine-grained Evaluation Framework For Diagnosing RAG - LlamaIndex

Pasted image 20250505103043.png

Day 2306 (25 Apr 2025)

Advanced Python features

https://blog.edward-li.com/tech/advanced-python-features/
- Protocols
- Python slots to make accessing fields of a class faster
- for-else statements to avoid if x is not None(): ...
- etc., TODO a lot of cool stuff
https://news.ycombinator.com/item?id=43770494

* did you know __init__.py is optional nowadays?

* you can do relative imports with things like "from ..other import foo"

* since 3.13 there is a @deprecated decorator that does what you think it does

* the new generics syntax also works on methods/functions: "def method[T](...)" very cool

* you can type kwargs with typeddicts and unpack: "def fn(*kwargs: Unpack[MyKwargs])"

* dataclasses (and pydantic) support immutable objects with: "class MyModel(BaseModel, frozen=True)" or "@dataclass(frozen=True)"

* class attributes on dataclasses, etc. can be defined with "MY_STATIC: ClassVar[int] = 42" this also supports abstract base classes (ABC)

* TypeVar supports binding to enforce subtypes: "TypeVar['T', bound=X]", and also a default since 3.13: "TypeVar['T', bound=X, default=int]"

* @overload is especially useful for get() methods to express that the return can't be none if the default isn't None

* instead of Union[a, b] or Optional[a] you can write "a | b" or "a | None" nowadays

* with match you can use assert_never() to ensure exhaustive matching in a "case _:" block

* typing has reveal_type() which lets mypy print the type it thinks something is

* typing's "Self" allows you to more properly annotate class method return types

* the time package has functions for monotonic clocks and others not just time()

Day 2305 (24 Apr 2025)

Ignoring stuff in flake8 pylint black etc.

Ignoring files:

# type: ignore
# flake8: noqa
# pylint: skip-file

vscode and cursor IDE settings

[]

Autosave and format

    "files.autoSave": "onFocusChange",
    "[python]": {
        "editor.formatOnSave": true,
        // "editor.defaultFormatter": "charliermarsh.ruff",
		"editor.defaultFormatter": "ms-python.black-formatter",
		// reformat everything w/ ruff
        "editor.codeActionsOnSave": {
            "source.fixAll": "explicit",
            "source.organizeImports": "explicit"
        },
    },

Misc

    "editor.rulers": [
        78, 88
    ],
	"editor.lineNumbers": "relative",
	"editor.formatOnPaste": true,
    "editor.formatOnSave": true,  // non-python stuff like settings.json

    "debug.allowBreakpointsEverywhere": true,
    "debug.inlineValues": "on",

Vim

<S-j>/<S-k> to switch tabs

    "vim.normalModeKeyBindingsNonRecursive": [
        {
            "before": [
                "J"
            ],
            "after": [],
            "commands": [
                "workbench.action.nextEditor"
            ]
        },
        {
            "before": [
                "K"
            ],
            "after": [],
            "commands": [
                "workbench.action.previousEditor"
            ]
        }
    ],

    //"editor.fontFamily": "'Droid Sans Mono', 'monospace', monospace",
    "editor.fontFamily": "Fira Code",
    "editor.fontLigatures": true,

<C-P> to show comman palette a la Obsidian

    "vim.normalModeKeyBindingsNonRecursive": [
        {
            "before": [
                "<C-p>"
            ],
            "commands": [
                "workbench.action.showCommands"
            ]
        }
    [...]
    "vim.visualModeKeyBindingsNonRecursive": [
        {
            "before": [
                "<C-p>"
            ],
            "commands": [
                "workbench.action.showCommands"
            ]
        }
    ],
    "vim.insertModeKeyBindings": [
        {
            "before": [
                "<C-p>"
            ],
            "commands": [
                "workbench.action.showCommands"
            ]
        }
    ],

Etc

w- “Cursor Dark High Contrast” is a really nice high-constrast theme!

    "editor.fontWeight": "bold",

Day 2296 (15 Apr 2025)

Evaluating NLG

Papers

[2303.16634] G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment ¹
- Excellent
- Main pitch:
  - “Auto CoT evaluation”: make the LLM generate a chain-of-thoughts based on provided criteria
  - get the LLM evaluate texts based on that CoT as form-filling task
  - weight by the probabilities of the votes as given by the LLM
    - 1*chance-of-1+2*chance-of-2
- Compares own evaluators to multiple others
[2302.04166] GPTScore: Evaluate as You Desire²
- basics:
  - task: e.g. summarize
  - aspect: e.g. relevance
  - eval protocol: how likely it is that this text was generated given this task and this aspect?
- main pitch:
  - LLM-estimated likelihood of text based on task+aspect+result.
  - For summary: {Task_Specification} {Aspect_Definition} Text: {Text} Tl;dr: {Summ}
- aspects:
UniEval [2210.07197] Towards a Unified Multi-Dimensional Evaluator for Text Generation³
- one trained evaluator to rule them all, based on boolean Q/A
- “is this a coherent summary?” -> chances for yes/no -> profit
LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models⁴

Approaches

Main goal: coherence with human judgement

Kinds

Reference-free

LLM-estimate the probability is a generated text
- assumption being that better texts have better probability
- downsides: unreliable / low human correspondence according to ¹
Ask an LLM how coherent / grammatical / criterium_name the output is from 0 to 5
- downsides: LLMs will often pick 3 but G-Eval works around that

Reference-based

BLEU/ROUGE and friends (n-gram based)
- low correlation with human metrics
(semantic) Similarity between two texts based on embeddings (BERTScore/MoverScore)
GPTScore²: higher probability to texts better fitting the framing
FACTUAL CONSISTENCY of generated summaries: FactCC, QAGS⁵

Framings (for GPT/LLM-based metrics)

Form-filling (G-Eval)
Conditional generation (=probability): GPTScore

Pairwise comparisons

Pairwise comparisons⁴

How to evaluate evaluations

Meta-evaluators

Meta-evaluators are a thing! (= datasets based on human ratings)
- G-Eval¹ uses
  - SummEval ⁶
  - TopicalChat⁷
  - QAGS⁵ evaluating factual consistency/hallucinations
- ¹
Correlation with human metrics

Stats

Many papers use various statistical bits to compare them.
G-Eval (p. 6)
- Spearman, Kendall-Tau correlations
- Krippendorf’s Alpha

On a specific task

Task formulation

Input:
- Advert for a car.
  
  Car has 4 wheels
- Target profile of possible client
  
  family with 10 kids 5 dogs living in the Australian bush
Output:
- Advert targeted for that profile:
  
  ROBUST car with 4 EXTRA LARGE WHEELS made of AUSTRALIAN METAL able to hold 12 KIDS and AT LEAST 8 DOGS

Concrete approaches

Factual correctness / hallucinations
- Framed as summarization evaluation: FactCC, QAGS⁵
GPTScore, G-Eval based on yet-to-be-determined-criteria
- Coherence, consistency, fluency, engagement, etc. — the criteria used by meta-evaluators?
- GPTScore aspects
- TODO: ask the car company what do they care about except factual correctness and grammar
information density-ish? Use an LLM to get key-value pairs of the main info in the advert (number_of_wheels: 4), formulate questions based on each, and score better the adverts that contain answers to more questions!
- Inspired by QAGS

TODO

LLM-as-a-judge and comparative assestment
think about human eval + meta-eval
think about what we want from the car company
- metrics they care about

G-Eval: <_(@liuGEvalNLGEvaluation2023) “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment” (2023) / Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, Chenguang Zhu: z / http://arxiv.org/abs/2303.16634 / 10.48550/arXiv.2303.16634 _> ↩︎ ↩︎ ↩︎ ↩︎
<_(@fuGPTScoreEvaluateYou2023) “GPTScore: Evaluate as You Desire” (2023) / Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, Pengfei Liu: z / http://arxiv.org/abs/2302.04166 / 10.48550/arXiv.2302.04166 _> ↩︎ ↩︎
<_(@zhongUnifiedMultiDimensionalEvaluator2022) “Towards a Unified Multi-Dimensional Evaluator for Text Generation” (2022) / Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, Jiawei Han: z / http://arxiv.org/abs/2210.07197 / 10.48550/arXiv.2210.07197 _> ↩︎
<_(@liusieLLMComparativeAssessment2024) “LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models” (2024) / Adian Liusie, Potsawee Manakul, Mark J. F. Gales: z / http://arxiv.org/abs/2307.07889 / 10.48550/arXiv.2307.07889 _> ↩︎ ↩︎
<_(@wangAskingAnsweringQuestions2020) “Asking and Answering Questions to Evaluate the Factual Consistency of Summaries” (2020) / Alex Wang, Kyunghyun Cho, Mike Lewis: z / http://arxiv.org/abs/2004.04228 / 10.48550/arXiv.2004.04228 _> ↩︎ ↩︎ ↩︎
<_(@fabbriSummEvalReevaluatingSummarization2021) “SummEval: Re-evaluating Summarization Evaluation” (2021) / Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, Dragomir Radev: z / https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00373/100686/SummEval-Re-evaluating-Summarization-Evaluation / 10.1162/tacl_a_00373 _> ↩︎
<_(@gopalakrishnanTopicalChatKnowledgeGroundedOpenDomain2023) “Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations” (2023) / Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, Dilek Hakkani-Tur: z / http://arxiv.org/abs/2308.11995 / 10.48550/arXiv.2308.11995 _> ↩︎