Image-2-Text

Numerical printout of brightness values, hand colored with pastel crayons.

In 1965, a camera system designed by NASA/JPL, thanks in part to Kirsch’s early experiments, flew to Mars to capture the first ever digital image of another planet. After receiving the first snippets of data, the team at JPL used a pastel set from a nearby art supply store to hand-color a numerical printout of the raw pixel values. At the time this was much faster than waiting for the image data to be computer processed. The results were a melding of human and machine, with little relation to its photographic counterpart.

First Image of Mars - Comparison between the handrawn vs. digital photo

I’ve always been very interested in space. Growing up I would crawl local libraries for interesting texts about the universe. Mostly for the artist renderings of far off worlds. When I first stumbled upon the Mariner 4 drawing above I made an instant connection to digital images and art that I never had put together before. This led me down the rabbit hole in terms of all this research. What better place to start than by reverse engineering what I believe the image compression was doing back in 1964.

For the purposes of this text I will be using the programming language Processing. Initiated in 2001 by Casey Reas and Benjamin Fry, Processing is an open-source programming language for artists. I will be assuming you’ve familiarized yourself with some of its concepts for this next part.

---

The Code

Processing

 import drop.*; 
 /////////////////////////////////////////////////  
 // Brandon A. Dalmer - 2023 
 // Image transcode - image to text - 82 X 110 pixels = 36" X 48" 
 /////////////////////////////////////////////////  
  
 File myFile; 
 SDrop drop; 
 PImage file; 
 float code = 0; // color tint 0-255 
 PrintWriter output; 
  
 /////////////////////////////////////////////////  
 void setup(){ 
   size(250,250); 
   textSize(32); 
   textAlign(CENTER); 
   fill(0); 
   drop = new SDrop(this); 
   text("DRAG 2 TEXT", width/2, height/2); 
 } 
  
 ///////////////////////////////////////////////// GENERATE TEXT 
 void draw(){ 
   if(file != null){ 
     output = createWriter("Mariner_4_output"+month()+year()+".txt"); 
     file.loadPixels(); 
     for(int y = 0; y < file.height; y++){ 
       for(int x = 0; x < file.width; x++){ 
         int loc = x + y*file.width; 
         float r = red(file.pixels[loc]); 
         float g = green(file.pixels[loc]); 
         float b = blue(file.pixels[loc]); 
         code = (r+g+b)/3; // B&W average 
         println(code); 
         String txt = nf((int)code, 3); 
         output.print(txt+" "); 
       } 
       output.println(); 
       output.println(); 
     } 
     output.flush(); 
     output.close(); 
     exit(); 
     } 
 } 
  
 ///////////////////////////////////////////////// LISTEN FOR DROP 
 void dropEvent(DropEvent theDropEvent){ 
   if(theDropEvent.isImage()){ 
     file = theDropEvent.loadImage(); 
   } 
 } 

P5.js

///////////////////////////////////////////////// 
// Brandon A. Dalmer - 2023
// Image transcode - image to text - 82 X 110 pixels = 36" X 48"
///////////////////////////////////////////////// 

let drop;
let img;
let code = 0;
let output;

function setup() {
  createCanvas(250, 250);
  textSize(32);
  textAlign(CENTER);
  fill(0);
  text("DRAG 2 TEXT", width / 2, height / 2);

  drop = new FileDrop(canvas, imageDropped);
}

function imageDropped(file) {
  if (file.type === 'image') {
    img = createImg(file.data, ''); // Load the image
    img.hide(); // Hide the image element
    img.size(width, height);
    img.parent(canvas);

    output = createWriter("Mariner_4_output" + month() + year() + ".txt");

    img.loadPixels();
    for (let y = 0; y < img.height; y++) {
      for (let x = 0; x < img.width; x++) {
        let loc = x + y * img.width * 4;
        let r = img.pixels[loc];
        let g = img.pixels[loc + 1];
        let b = img.pixels[loc + 2];
        code = (r + g + b) / 3; // B&W average
        println(code);
        let txt = nf(int(code), 3);
        output.print(txt + " ");
      }
      output.println();
      output.println();
    }
    output.flush();
    output.close();
  }
}

function draw() {
  // Your drawing code here
}

HTML

<!DOCTYPE html>
<html>

<head>
  <style>
    body {
      background-color: black;
      color: white;
    }
  
    /* Style the "Choose File" input element */
    #fileInput {
      background-color: #4CAF50; /* Background color */
      color: white; /* Text color */
      padding: 10px; /* Padding around the text */
      border: none; /* Remove the border */
      cursor: pointer; /* Change cursor to a pointer on hover */
      font-size: 30px;
    }
    
    #output {
      padding-top: 10px;
      text-align: center;
      font-size: 30px;
    }
  </style>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.4.0/p5.js"></script>
</head>

<body>
  <p>BRANDON A. DALMER 2023 // IMAGE-2-TEXT</p>
  <input type="file" id="fileInput" accept="image/*">
  <p>CLICK TO UPLOAD IMAGE // width must be less than: 75px</p>
  <div id="output"></div> <!-- Create a new div to display pixel values -->
  <canvas id="canvas"></canvas>

  <script>
    let img;
    let outputDiv;

    function setup() {
      createCanvas(windowWidth, windowHeight);
      background(0);
      outputDiv = document.getElementById("output"); // Get the output div

      // Add a change event listener to the file input element
      document.getElementById("fileInput").addEventListener("change", handleFileSelect);
    }

    function handleFileSelect(evt) {
      outputDiv.innerHTML = ""; // Clear the previous content
      let file = evt.target.files[0];
      if (file.type.match("image.*")) {
        img = loadImage(URL.createObjectURL(file), function(loadedImg) {
          if (loadedImg.width <= 75) { // Check if the image has the required width
            clear();
            background(0);
            image(loadedImg, 10, 10, img.width * 10, img.height * 10); // Draw the image on the canvas
            loadPixels();
            let pixelValues = [];
            
            for (let y = 0; y < loadedImg.height; y++) {
              let line = [];
              for (let x = 0; x < loadedImg.width; x++) {
                let pixelColor = loadedImg.get(x, y); // get color of the pixel
                let r = red(pixelColor);
                let g = green(pixelColor);
                let b = blue(pixelColor);
                let code = (r + g + b) / 3;
                let intCode = int(code);
                let formattedValue = nf(intCode, 3);
                fill(formattedValue);
                line.push(`<span style="color:rgb(${r},${g},${b})">${formattedValue}</span>`);
              }
              pixelValues.push(line.join(" ")); // join the line and add it to the pixelValues array
              line = []; // clear the line for the next row
              pixelValues.push(line.join(" ")); // join the line and add it to the pixelValues array
            }

            // Set the font size for the entire outputDiv using CSS
            outputDiv.innerHTML = pixelValues.join("<br>");
            outputDiv.innerHTML += "<br>0 - 28 // BLK | 29 - 57 // NG1 | 58 - 86 // NG2 | 87 - 115 // NG3 | 116 - 144 // NG4 | 145 - 173 // NG5 | 174 - 202 // NG7 | 203 - 231 // NG8 | 232 - 255 // WHT";
            outputDiv.style.fontFamily = "Courier New"; // set font to Courier New
            outputDiv.style.fontWeight = "bold"; // set text to bold
            outputDiv.style.fontSize = "0.50vw"; // Set a specific font size if needed
          } else {
            alert("Image width must be 75px !!");
          }
        });
      } else {
        alert("Unsupported file type: " + file.type);
      }
    }
  </script>
</body>

</html>

Global

code - this will act as the output number used to calculate brightness of each pixel. 0 meaning black, 255 meaning white.

file - this program uses the drop library. A file can be any size, however since we’re going pixel by pixel here a very low dpi image should be used. An example being an image no larger than 82 pixels by 110 pixels.

---

Draw

This program loads our file as a set of pixels. One by one it will cycle through its color value, i.e (RGB) 255, 0, 0 for pure red. After grabbing these values it will then average them.

(Red + Green + Blue) / 3 = average value

These values are then recorded before moving on to the next pixel. Once completed, our values are turned back into a text document which gets saved. The smaller the image used the faster the process. This program is almost instantaneous. The new file is then saved in the sketch folder of the program.

Painting

Certain acrylic paint colors straight out of the tube have a tonal value very close to black. By thinning this paint down in stages we can form a tonal gradient. This is best done with an Acrylic medium like GAC 100 or Acrylic Glazing Medium. The former has a drying retarder which will for an extended working time. Some ideal "black" colors are:

• Alizarin Crimson

• Permanent Maroon

• Dioxazine Purple

• Anthraquinone Blue

• Prussian Blue

• Phthalo Blue (RS/GS)

• Phthalo Green (BS/YS)

• Sap Green

• Quin Nickel Azo

• Paynes Gray

• Bone Black

In order to have a complete tonal value we will need to thin one of the above colors down into 9 diffrent tones. Once this is done, the numbers generated by this program can be matched to each tone. These are as follows:

BLK 000-025 / N2 026-051 / N3 052-077 / N4 078-103 / N5 104-129

N6 130-155 / N7 156-181 / N8 182-207 / N9 208-233 / WHT 234-255

Here are some examples of this process:

AS IC AN, Acrylic on Canvas 48" X 72", 2023