Turning Camera Images into Words Helps 3D Scans Understand Objects Without Training

From photographs to point clouds through natural language

The starting point is a set of ordinary photographs and a matching 3D scan. The cameras are already positioned, their lenses calibrated, and the geometry between image pixels and space is known. No human has to label points in the cloud as brick, crack, or column. No model is trained on that particular building or room.

A vision-language model looks at each photograph and writes a dense description of what it sees. It can name objects, parts, materials, colors, textures, and damaged conditions. A second language model then turns that free text into compact records. Each record has a category, a short label, a prompt for segmentation, and optional attributes. Defects are kept separate from normal surfaces.

Those words must be tied to pixels. An off-the-shelf open-vocabulary 2D segmenter reads the prompts and produces masks in the image. A mask can cover a column, a crack on that column, or a patch of weathered stone. Masks may overlap because one surface can carry several meanings at once.

The system then moves from pixels to points. Using camera poses and intrinsics, each 3D point is projected into nearby views where it should be visible. Views blocked by closer geometry are ignored. The language records attached to the masks become sentence embeddings in a general-purpose text space. A point therefore receives meaning through words, not through the shared image-and-text space that many earlier systems use.

All components are frozen off-the-shelf models used at inference time. The description model writes, the structuring model organizes, the segmenter grounds, and the sentence-embedding model turns phrases into vectors. Nothing is trained on the target scene or domain.

Weighting evidence from many cameras before averaging each point’s meaning

A single point in a 3D scan can be seen from many cameras. Each camera view may describe the same surface differently. One photo might show a column clearly, while another catches only its edge. Before those descriptions are averaged, the process asks how trustworthy each view is. Close views count more than distant ones. A point near the image center counts more than one at a blurry edge. Sharp pixel masks count more than rough ones. If another closer surface hides the point, that camera is ignored. This keeps the average from being dragged by weak evidence. The weighted descriptions are first combined into one mean vector for the point. Then views that sit far from that average are removed. The remaining views are averaged again. This step helps reject bad masks or invented details. In one heritage scene, a multi-vector average that respected scale reached 30.18 on an overlap score. The higher the score, the closer the result matches the correct answer. It kept several weighted descriptions instead of one blended vector. It beat a simpler average that removed distant views first, by 4.36 points. A soft-weighted version also did better, scoring 27.27 against 25.82. A method that simply picked the strongest view scored only 20.68. It ignored the weights and distorted the angle-based distances between sentence meanings. Those numbers show that not every camera view deserves equal power. Points with no usable views borrow from nearby points. The closer a neighbor is, the more influence it has. This fills gaps without inventing new evidence.

How one corrected word can reshape an entire heritage scan

A single word can move millions of points in a heritage scan. The fix here is almost embarrassingly small. A prompt that said “floor” was changed to “floor/grass.” That one edit improved overall accuracy by 10.28 points for MiniLM, the sentence-embedding model used for language meaning. CLIP, which blends image and text in a shared visual-language space, gained only 2.56 points from the same change. The score measures how well predicted labels overlap true labels. The difference is not a rounding error. It shows that the meaning of a word can travel through an entire scene.

Before the fix, 3.04 million true floor/grass points were called column by MiniLM. Why? “Floor” was too narrow for heritage surfaces. Many stone floors, worn paving, or ground-like areas did not match the plain word well. When the prompt became “floor/grass,” those points found a better linguistic home. 96.3% of them returned to floor/grass. That cleanup also helped columns. Column accuracy jumped by 29.00 points because fewer floor points contaminated it. Vault/ceiling stayed untouched. This is not random noise. It is semantic structure responding to wording.

A second edit made the same point. Changing “vault” to “vault/ceiling” added 2.93 points for MiniLM, but only 0.89 for CLIP. Arch accuracy rose by 5.56 points. Column barely moved. Again, the effect followed real architectural relationships. A ceiling-like surface is not a random neighbor of an arch. It belongs to the same structural logic. When both fixes were applied together, MiniLM gained 13.26 points while CLIP gained only 3.55. The lesson is simple: in language-based scanning, one corrected term can redraw boundaries across a whole heritage scene.

Separating real objects, absent objects, and nonsense queries in 3D space

A 3D scan can be asked questions in words. The hard part is telling three kinds of answers apart. One says an object is really there. Another says it is a normal thing but missing from this room. A third says something impossible or nonsense. If the system cannot separate these, it may point to a wall and call it a “fake ceiling”, or treat a missing sofa as if it were present.

The test used 25 object names outside the usual vocabulary of an indoor-scene benchmark. For each name, the system looked at the 100 most similar points in the scan. Similarity is a score. Higher means the point’s meaning matches the query better. One model turned words into general-purpose text meanings. It gave present objects a mean score of 0.704. Absent objects scored 0.561. Nonsense queries scored 0.469. The gaps matter more than the raw scores. Present beats absent by 0.143. Absent beats nonsense by 0.092. That is a clean ladder: real, plausible but missing, impossible.

A second model linked images and words in one shared space. It did not make that ladder. Its present score was 0.590, absent 0.529, and nonsense 0.522. Present still beat absent, but only by 0.061. Absent and nonsense were almost tied, separated by just 0.008. In other words, it could tell that a real object was there, but it struggled to say whether a missing thing was merely absent or simply nonsensical.

The result is not perfect. Four of the 25 classes inverted for each encoder. For those cases, an absent query scored slightly higher than a present one. “Fake ceiling” inverted for both systems. That suggests some scene assignments may confuse the test. Still, the overall pattern points to a practical difference. A text-focused meaning map can keep three meanings apart: seen here, not seen but possible, and not meaningful at all.

A language bridge for three-dimensional understanding

The main finding is that 3D points can carry meaning without a model trained on the scene. Images are turned into plain descriptions, those descriptions become sentence embeddings, and each point collects evidence from many views. This lets a system label 3D points with pretrained models that are not retrained. No human labels are needed for a new building or room. The system can answer questions about objects that were never part of a training list.

That is the opportunity. Heritage sites, interiors, and scanned environments become searchable by ordinary language. A crack on a column, a damaged vault, or an absent fixture can be located without teaching a network what those things look like in 3D. The approach also separates present objects from missing ones and from nonsense queries better than vision-language embeddings do. Precise wording matters. Small vocabulary fixes can change large areas of a scan.

The obstacles are real. The pipeline still needs a 3D scan, posed photos, camera calibration, and lens correction. It inherits errors from image description, segmentation, and language models. Ambiguous names hurt performance. Rare but valid observations can be mistaken for noise. Bigger vision-language models write richer descriptions, but they may exceed practical hardware limits. Some classes still confuse present with absent objects.

If a point cloud can be understood through the words we already use, what happens when those words become good enough to repair, monitor, and protect the places we built?