How a sentence becomes a vector in this project, what each symbol in the equations means, and what that vector can and cannot justify.
An embedding is a numeric fingerprint of a sentence: a fixed-length list of numbers. The model used here is paraphrase-multilingual-MiniLM-L12-v2 (Sentence-Transformers checkpoint; Reimers and Gurevych 2019), built from a twelve-layer MiniLM encoder (Wang et al. 2021) over a multilingual transformer stack (Vaswani et al. 2017). Text is split into WordPiece tokens (Devlin et al. 2019). Weights are frozen on the Montreal corpus (not fine-tuned on your documents). Implementation: SentenceTransformer.encode in scripts/run_semiosphere_feature_space.py.
Two fingerprints that sit close together usually share topic. They do not reliably preserve register (scare quotes, gathering-not-symposium, which shell a stretch belongs to). That is why the Lotman reading treats the encoder as an instrument on the primary language layer - good at translation-equivalence, weak at institutional shell by itself.
Take every token's internal vector from the last layer, add them up, divide by how many tokens there were. The result e is the sentence fingerprint stored for the probe and retriever.
| Symbol | Stands for | What happens to it |
|---|---|---|
| e | Sentence embedding | Built once per scored stretch; one row in the embedding matrix. |
| n | Token count | Longer sentences have larger n; each token counts equally in the average. |
| hi | Hidden vector for token i | Comes out of layer 12; already mixes context from the whole sentence. |
| Σ | Sum | Add all n token vectors component-wise (384 additions per dimension). |
| 1/n | Divide by count | Turns a sum into an average so length does not automatically scale the arrow. |
| ℝ384 | 384-dimensional space | Each e is a point (arrow from origin) with 384 coordinates. |
Toy example (2 numbers instead of 384): three tokens with hidden vectors (1,0), (1,2), (3,2). Sum = (5,4). Divide by 3 → e ≈ (1.67, 1.33). One sentence, one arrow.
The retriever and probe compare fingerprints by cosine similarity: how much two arrows point the same way, not how long they are (standard vector-space measure; Manning, Raghavan, and Schütze 2008; computed via scikit-learn in the probe script).
Multiply matching coordinates and add the products. Large positive value → many coordinates push together; mixed signs → directions disagree.
| Symbol | Meaning |
|---|---|
| x, y | Two sentence fingerprints (query vs candidate, or secret vs outward). |
| xk | The kth number in vector x (one coordinate). |
| x · y | A single score from the whole vector - not itself the cosine yet. |
2D example: x = (0.8, 0.6), y = (0.6, 0.8). Dot = 0.8×0.6 + 0.6×0.8 = 0.96.
Dot the vector with itself, take the square root - Pythagoras in 384 dimensions. Norm is how long the arrow is before you compare direction.
Cosine divides by ‖x‖ and ‖y‖ so a very long vector does not automatically look "more similar."
Before comparing secret and outward clouds in the probe, vectors pass VecMap-style preprocessing: L2 normalize, subtract the training-side mean, L2 normalize again (Artetxe, Labaka, and Agirre 2016; Artetxe, Labaka, and Agirre 2018; function artetxe_preprocess in run_semiosphere_feature_space.py). That is the centering and normalization step so cosine tracks angle cleanly. Cross-lingual maps on top of those vectors follow the Mikolov → Xing → Artetxe chain documented on Semiotic coordinate plain §5 (Mikolov, Le, and Sutskever 2013; Xing et al. 2015; orthogonal Procrustes: Schönemann 1966).
Montreal retriever: flat search returned the internal memo at cosines about 0.48-0.76 to outward-clip questions - high theme overlap, wrong shell. Scoped by shell → tier confusion 0.
Orange and blue share a topic direction (high cosine). Grey is perpendicular (cosine ≈ 0). Real work uses 384 dimensions; this is a teaching projection.
Multilingual training pulls translation mates together. Register is not an axis the frozen encoder labels for you.
Shared famine/film topic clumps; shell differs. Separation needs typing, scoped retrieval, or pair-read - not raw cosine across Russian and English alone.
| Question | Encoder alone | With typed archive |
|---|---|---|
| Same topic, different week? | Often yes (high cosine) | Pair-read dates the relay |
| Scare quotes stayed internal? | Partly (forbidden-operator probe) | L2 tensor + pair-read |
| Secret → embassy recoding? | Not a global map (Procrustes retraction) | Operator table, misses |
| Newspaper proving forgery? | Flat retrieval: memo wins (6/6) | Shell scope + halt: 0 tier confusion |
Related figures: 3D embedding + relays · Theme vs register (2D PCA) · Probe map Y ≈ W X · LLM theme vs shell bars
This appendix covers the encoding and similarity layer only; historical claims still require pair-read. Each step below maps to its source and to the lab script.
| Step on this page | What the project does | Derived from |
|---|---|---|
| Encoder checkpoint | paraphrase-multilingual-MiniLM-L12-v2, frozen | Reimers and Gurevych 2019; Wang et al. 2021 |
| Tokenization | Model tokenizer (WordPiece) | Devlin et al. 2019 |
| Contextual vectors hi | Last hidden layer, 12 blocks | Vaswani et al. 2017 |
| Mean pool → e | SentenceTransformer.encode default pooling | Reimers and Gurevych 2019 |
| Dot product & cosine | sklearn.metrics.pairwise.cosine_similarity | Manning, Raghavan, and Schütze 2008 |
| L2 → center → L2 | artetxe_preprocess on train/test/inventory | Artetxe, Labaka, and Agirre 2016; Artetxe, Labaka, and Agirre 2018 |
| Secret→outward map (not re-derived here) | Orthogonal Procrustes + ridge + controls | Schönemann 1966; Mikolov, Le, and Sutskever 2013; Xing et al. 2015; probe logic Hewitt and Manning 2019; controls Hewitt and Liang 2019 |
| Flat retrieval (E4) | Rank by cosine over pooled inventory | Project experiment log; failure mode, not a published method claim |
Lab trace: peter_htr_experiment/docs/EXPERIMENT_LOG_semiosphere_feature_space.md, 2026-09-10 literature section. Comps placement: reading list · comprehensive lists.
Alain, Guillaume, and Yoshua Bengio. "Understanding Intermediate Layers Using Linear Classifier Probes." International Conference on Learning Representations (ICLR), 2017. Workshop track. (Linear classifier probes on frozen representations.)
Artetxe, Mikel, Gorka Labaka, and Eneko Agirre. "Learning Principled Bilingual Mappings of Word Embeddings while Preserving Monolingual Invariance." In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2289-94. Austin, TX: Association for Computational Linguistics, 2016.
Artetxe, Mikel, Gorka Labaka, and Eneko Agirre. "A Robust Self-Learning Method for Fully Unsupervised Cross-Lingual Mappings of Word Embeddings." In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 789-98. Melbourne: Association for Computational Linguistics, 2018. (VecMap preprocessing recipe used in the probe script.)
Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, 4171-86. Minneapolis, MN: Association for Computational Linguistics, 2019.
Hewitt, John, and Christopher D. Manning. "A Structural Probe for Finding Syntax in Word Representations." In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, 4129-38. Minneapolis, MN: Association for Computational Linguistics, 2019.
Hewitt, John, and Percy Liang. "Designing and Interpreting Probes with Control Tasks." In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2733-43. Hong Kong: Association for Computational Linguistics, 2019.
Manning, Christopher D., Prabhakar Raghavan, and Hinrich Schütze. Introduction to Information Retrieval. Cambridge: Cambridge University Press, 2008. (Cosine similarity as a vector-space retrieval score.)
Mikolov, Tomas, Quoc V. Le, and Ilya Sutskever. "Exploiting Similarities among Languages for Machine Translation." arXiv, 1309.4168 (2013). (Linear map between embedding spaces.)
Reimers, Nils, and Iryna Gurevych. "Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks." In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3982-92. Hong Kong: Association for Computational Linguistics, 2019.
Schönemann, Peter H. "A Generalized Solution of the Orthogonal Procrustes Problem." Psychometrika 31, no. 1 (1966): 1-10.
Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. "Attention Is All You Need." In Advances in Neural Information Processing Systems 30, 5998-6008. 2017.
Wang, Wenhui, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. "MiniLM: Deep Self-Attention Distillation for Task-Agnostic Knowledge Distillation of Transformer Encoders." In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 5770-81. Online: Association for Computational Linguistics, 2021.
Xing, Chao, Dong Wang, Chao Liu, and Yiye Lin. "Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation." In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1006-11. Denver, CO: Association for Computational Linguistics, 2015.
Dotted words in the text open a short definition; click again or use Full definition ↓ to jump here.