Koke_Cacao – FinalDocumentation



Writing Minecraft Shader

Final Image of In-Progress Minecraft Shader
Final Image of In-Progress Minecraft Shader

What is Shader?

Use Case

// TODO: case study


Shader, roughly speaking are functions that transform geometry to pixel on screen. There are many things people called shader. For example, in Unity, shader is defined:

.shader file

  • properties: customizable user input (texture, normal, …)
  • hidden properties: transformation matrix, mesh
  • shader: a combination of shaders for different platform
    • subshader #1: high quality render use this
      • .vsh
      • .fsh
    • subshader #2: low quality render use this

This part is inspired by Freya Holmér’s Video.

For consistency, when the word shader is mentioned, we refer a combination of files (that we called shader programs) that looks like .vsh and .fsh.

General Shader Pipeline with Vertex and Fragment Shader

data = vertices_of_geometry_in_world
data = data.map(vsh)
data = rasterization(data)
data = data.map(fsh) # actually this is in the process of rasterization as depth test is after fsh
pixels = antialiasing(data)

# This is a vertex buffer file written in python style pseudo code
def vsh(aPos, aNormal, aColor, aUVCoord, ... you define):
  # layout (location = 0) in vec3 aPos;
  # layout (location = 1) in vec3 aNormal;
  # layout (location = 2) in vec3 aColor;

  # uniform mat4 model;
  # uniform mat4 view;
  # uniform mat4 proj;
  global model
  global view
  global proj

  # DO MATH HERE: move vertex position, add attributes to vertex

  # out vec3 crntPos;
  # out vec3 normal;
  # out vec3 color;
  # Notice [gl_Position] here is not specified. We must use this name to tell OpenGL that this is the vertex position, not attributes we defined
  # crntPos is passed for light calculation
  return gl_Position, crntPos, normal, color, ... you define

def rasterization(all_vertice):
  # magic interpolation of all other attributes based on [all_vertice[gl_Position]]
  return fragments_and_interpolated_attributes_from_vertices

def fsh(crntPos, normal, color):
  # in vec3 crntPos;
  # in vec3 normal;
  # in vec3 color;

  # uniform vec4 lightColor;
  # uniform vec3 lightPos;
  # uniform vec3 cameraPos;
  global lightColor; # assuming there is one light
  global lightPos; # assuming there is one light
  global cameraPos;

  # DO MATH HERE: light, shadow, special effect

  # out vec4 FragColor;
  return FragColor

The concept of fragment is nearly identical to pixel, except multiple fragments can contribute to one pixel. This is so that we can implement multisampling or antialiasing. If no multisampling nor antialiasing, then there is 1-to-1 correspondence between fragment and pixel on screen.

Shader Pipeline with Vertex, Geometry, and Fragment Shader

The idea is basically the same except we add a geometry shader between vertex and fragment shader.

data = vertices_of_geometry_in_world
data = data.map(vsh)
data = assemble_to_triangles(data) # returns a list of triangles, lines, or points
data = data.map(gsh)
data = rasterization(data)
data = data.map(fsh) # actually this is in the process of rasterization as depth test is after fsh
pixels = antialiasing(data)

def vsh(...):
  # Everything is the same except we do not do transform and projection any more. They are done in gsh now.

def gsh(crntPos, normal, color):
  # layout (points) in;
  # layout (triangle_strip, max_vertices=8) out;
  setInputLayout(points); # Input can only be: points (one point), lines, triangles, lines_adjacency, triangles_adjacency
  setOutputLayout(triangle_strip, max_vertices=8); # Output can only be: points (multiple points), line_strip (multiple lines), triangle_strip (multiple triangles). We want to output vertices at maximum.

  # in vec3 crntPos[];
  # in vec3 normal[];
  # in vec3 color[];

  # some uniform

  # DO MATH HERE: remove triangle, add mesh to point, add more triangles near this triangle, add or remove attributes to vertex
  vec4 vertexPos = gl_in[0].gl_Position; # for example, get the position of first vertex

  # Now we create a rectangle consists of 4 vertice
  # Remember to transform and project those vertices
  gl_Position = vec4(...);
  gl_Position = vec4(...);
  gl_Position = vec4(...);
  gl_Position = vec4(...);

  # We also want to create a line
  gl_Position = vec4(...);
  gl_Position = vec4(...);

  # out vec3 cat;
  return cat

def fsh(cat):
  # in vec3 cat;

This section is inspired by ThinMatrix’s OpenGL Tutorial 49: Geometry Shader Introduction.

About Minecraft and Minecraft Rendering

Minecraft is a popular sandbox survival game and I have a long relationship with this game. Funny enough, my first line of code in a general purpose language environment is a hello world printed in the Minecraft server console, and I have since then built a commercial Minecraft (with my costom GTA mode) server hosted about 300k players. To me and most of Minecraft enthusiasms, Minecraft is more than a game as it hosts communities of different interests: creative map makers, adventure map makers, minigame designers, traditional survival players, redstone technicians, youtubers, pvp players, pve players, community builders.

Now, Minecraft shaders is a program to change how the traditional Minecraft world looks like by taking over the shader engine that is used to render Minecraft. It is a great way to practice glsl skill and learn computer graphics because:

  1. Minecraft is exciting. It encourage you to code.
  2. You don’t have to play with testing geometry, which is boring. Minecraft provides you with a full game you can test on. It lets you build things gradually from easy to hard. You will be challenged with: terrain, lighting, shadow, transparent block, cloud, changing of the sun, moon phases, animals, water, reflective armor, rain, thunder storm, beacon special effect, posion special effect, different biomes…
  3. Minecraft is beyond toy environment. It provides you with an overview of how actual game rendering is pipelined. It has multiple stages (more than .vsh, .fsh), use differed rendering, and dynamic load of buffers.
  4. There are existing communities for Minecraft Shader development. Join Shader Labs
  5. You don’t need to worry about getting attributes from Geometry. OptiFine, a mod that optimizes Minecraft renders provides with you many attributes you can use for free. You can find OptiFine Documentation Here.

However, there is one downside of learning shader using Minecraft: the OpenGL language version is quiet old. Quote from Shader Lab: “Anything that says #version 120 will be best. Minecraft 1.17 will use #version 150, but you are not restricted to just these specific #version’s. You can use any #version that your GPU/drivers support.”

OpenGL is a graphics engine that provides basic structure of rendering pipeline. Its main job is to do basic geometry load and transform, compile shader, rasterization, and talk with GPU.

Since Minecraft’s rendering pipeline is way more complex than a simple shader toy example, we will use some specialized terminology:

  • stage: there are 4 possible stages for each program: .vsh, .gsh, .fsh, .csh.
  • shader program: refer to shadows, gbuffers, composites, deferred, ... one each contains a collection of stages (.vsh, .gsh, .fsh, .csh)
  • pipeline: a collection of shader programs
  • pass: general term refer to compute from something to every pixel, filling the entire screen space without leaving out any pixel blank (e.g. without distinguish between entities and blocks)

Rendering Pipeline: Deferred Rendering, Pipelines, Stages, and Buffer

Deferred Rendering

Naive Explanation

// TODO: remove naive explanation

Forward Rendering: for visible fragment (regardless overlaps), calculate light. We do calculation for every point of geometry surface in the pyramid volume (clip space) before projection to the screen.

for (fragment in models):
  for (lightbulb in lights):
    color += calculateLight(fragment, lightbulb)
Color, Depth, and Normal buffers, each of size klzzwxh:0008 (Images by astrofa, via Wikimedia Commons.)
Color, Depth, and Normal buffers, each of size W×H×3 

(Images by astrofa, via Wikimedia Commons.)

Final lighting (shading) result generated using the three buffers. (Image by astrofa, via Wikimedia Commons.)
Final lighting (shading) result generated using the three buffers. (Image by astrofa, via Wikimedia Commons.)

Deferred Rendering: We don’t need to calculate light for unseen part of the surface. However, as a trade off, we need to reconstruct the 3D world by inverse projection from screen space and calculate the light for every reconstructed fragment.

for (fragment in models):
  albedo = get_albedo(fragment)
  normal = get_normal(fragment)
  depth = get_depth(fragment)
for (pixel in screen):
  for (lightbulb in lights):
    color += calculateLight(pixel, lightbulb, albedo, normal, depth)

Complex Explanation

The idea is that we don’t want to calculate lights for surfaces that is not visible to the camera. However, traditionally, depth test is done after fragment shader (for pixel, replacing color if smaller depth), and therefore fragment shader is run for every surface inside clip space.

Depth testing is done in screen space after the fragment shader has run. Today most GPUs support a hardware feature called early depth testing. Early depth testing allows the depth test to run before the fragment shader runs. (Depth testing)

However, as Nicol Bolas indicated in Stackoverflow, early depth test will not guarantee non-visible fragments to be passed to fragment shader.

Therefore, we need to do 2 passes: first render depth to rasterized fragments, and then assemble to pixels.

while(...) { // render loop
  // 1. geometry pass: render all geometric/color data to g-buffer
  glBindFramebuffer(GL_FRAMEBUFFER, gBuffer);
  glClearColor(0.0, 0.0, 0.0, 1.0); // keep it black so it doesn't leak into g-buffer
  for(Object obj : Objects) {
  // 2. lighting pass: use g-buffer to calculate the scene's lighting
  glBindFramebuffer(GL_FRAMEBUFFER, 0);

Pseudocode borrowed from Deferred Shading.

Pipelines, Stage and Buffer


Stages are piece of code in a file that takes in either geometry, fragment, (and uniform, buffer), output things required by other stages. It is often run on every vertex or fragment. In a typicall simple shader, vertex shader (.vsh) and fragment shader (.fsh) are two stages. But in Minecraft, there are more stages.

In OptiFine render, we have many many programs each have at maximum 4 stages.

Four Stages in OptiFine:

  • vertex stage (.vsh):
    • run for: every vertex
    • input: vertex attributes (position, color, light level)
    • output: projected vertex position and attributes if geometry stage does not exists // QUESTION: what if exist
  • geometry stage (.gsh): Optional // TODO: experiment with it
  • fragment stage (.fsh): // TODO: documentation says it is after rasterization. Really?
  • compute stage (csh): Optional. It does not know about any geometry in the world, but can write directly to a buffer at any location. (Normally when a fragment stage writes to a buffer, it is restricted to only writing at the location of the pixel it was assigned to.) // TODO: try it

We transfer interpolated data by using varying.


Programs in Pipelines:

  • Shadows (shadow.fsh/vsh): suppose to project the world to the sun (a camera looking at player from the sun). We need its depth map (z-buffer) for calculating shadow.
  • Gbuffers (files starting with gbuffers_): render terrain, entities, sky (later than shadow, in order or Skybasic -> skytextured -> terrain (opaque blocks, wind effect) -> tile entities (entities and entity blocks) -> textured, textured_lit (particles) -> deferred -> weather)
  • Composites (composite(N) or final): run after all geometry (all the gbuffers). For post-processing effects: lighting, ambient, occlusion, fancy clouds, reflections, refractions… You can write to as many buffer(s) as you want, with whatever data you want.
  • Deferred: (deferred(N).fsh/vsh): similar to the composite programs, but runs in the middle of terrain rendering instead of after it. More specifically, they run after all opaque objects have rendered, and before any transparent objects. There is no real goal here. You can write to as many buffer(s) as you want, with whatever data you want.
Program Name Rendered Geometry Fallback
Shadow Map
shadow all none
Shadow Composite
shadowcompshadowcomp<1-15> none
prepareprepare<1-15> none
gbuffers_basic leash none
gbuffers_line block selection, fishing line gbuffers_basic
gbuffers_textured particles gbuffers_basic
gbuffers_textured_lit lit/emissive particles, world border gbuffers_textured
gbuffers_skybasic sky, horizon, stars, void gbuffers_basic
gbuffers_skytextured sun, moon gbuffers_textured
gbuffers_clouds clouds gbuffers_textured
gbuffers_terrain opaque geometry (including cutout transparency) gbuffers_textured_lit
gbuffers_damagedblock damaged block overlay gbuffers_terrain
gbuffers_block block/tile entities gbuffers_terrain
gbuffers_beaconbeam beacon beam gbuffers_textured
gbuffers_entities entities gbuffers_textured_lit
gbuffers_entities_glowing glowing entities (spectral effect) gbuffers_entities
gbuffers_armor_glint armor glint overlay gbuffers_textured
gbuffers_spidereyes eyes of spiders, endermen and enderdragons gbuffers_textured
gbuffers_hand hand, opaque handheld items gbuffers_textured_lit
gbuffers_weather rain, snow gbuffers_textured_lit
deferreddeferred<1-15> none
Translucent G-Buffers
gbuffers_water translucent geometry gbuffers_terrain
gbuffers_hand_water translucent handheld items gbuffers_hand
compositecomposite<1-15> none
final none


Buffer (Framebuffer Attachments) are memory shared accross different program (with special permission: some earlier stage cannot access buffers only created for later stage). This is especially useful for deferred rendering. For example, any program executed after shadow program can then access depth buffer created by shadow program.

Quote from Shader concepts by BUILDERB0Y: Create 2 buffers. one is a material buffer, the other is the translucent buffer. Make all transparent objects output their color to the translucent buffer, and a number representing their ID (passed in with varyings) to the material buffer. Composite can read the material buffer, and mix the translucent buffer with the opaque color buffer differently depending on the ID. This will allow effects such as fog behind water, or only applying reflections to glass and water but not slime blocks or nether portals. As you may have guessed though, the material buffer can only store one ID per pixel. In most cases, the ID will be that of the closest transparent object to the camera. Everything behind it will be ignored. This means that if you look at water through stained glass, suddenly it won’t have thick blue fog anymore. A lot of shader packs have similar issues. Sadly, there’s no easy way to fix this. Still, this should give you an idea of what is and isn’t possible to do with OptiFine’s pipeline.

Implementing The Shader

Default Minecraft Looking
Default Minecraft Looking

Understand Deferred Rendering

So how to write Minecraft shader? Minecraft already has its shader. Instead of adding, we replace them. We first play with final pass final.vsh and final.fsh.

We play with vertex shader first.

#version 140
varying vec4 texcoord;
void main() {
  gl_Position = gl_ModelViewProjectionMatrix * (gl_Vertex + vec4(vec2(0.1f, 0.25f), 0.0f, 0.0f));
  texcoord = gl_MultiTexCoord0; // vec4 of texture output
Shifting Screen Vertex
Shifting Screen Vertex

We notice that the entire screen is shifted. This is a feature of deferred rendering. The vertex shader does not take in actual vertice, but rather takes a simple plane.

#version 140
varying vec4 texcoord; // screen coord from 0 to 1
uniform sampler2D gcolor; // texture storing screen color
void main() {
  vec4 albedo = texture2D(gcolor, texcoord.st);
  gl_FragColor = albedo;

In fragment shader, we simply pass albedo from the buffer to the next buffer gl_FragColor. Since it is final, the screen will directly output this color as pixel color.

Implement Ambient Light and Color

Time to implement light. We need to get the lightmap (as well as normals, colors, and texture coordinate) from vertex shader gbuffers_terrain.vsh and pass into gbuffers_terrain.fsh

varying vec4 texcoord;
varying vec3 normal;
varying vec4 color;
varying vec2 lightmapCoords;
void main() {
  gl_Position = ftransform();
  texcoord = gl_MultiTexCoord0;
  normal = gl_NormalMatrix * gl_Normal;
  color = gl_Color;
  lightmapCoords = (lightmapCoords * 33.05f / 32.0f) - (1.05f / 32.0f);

In fragment shader gbuffers_terrain.fsh, we simply store them in gbuffer for deferred rendering. The comment /* RENDERTARGETS: 0,1,2 */ tells OptiFine that buffer position in our code 0, 1, 2 correspond to actual buffer position 0, 1, 2. This is so that OptiFine does not need to attach other buffers that we don’t use.

varying vec4 texcoord;
varying vec3 normal;
varying vec4 color; // for biome specific color and ambient occlusion
varying vec2 lightmapCoords;

uniform sampler2D texture;

void main() {
  vec4 albedo = texture2D(texture, texcoord.st) * color;
  /* RENDERTARGETS: 0,1,2 */
  gl_FragData[0] = albedo;
  gl_FragData[1] = vec4(normal * 0.5f + 0.5f, 1.0f); // normal is -1 to 1. We need to fit in 0 to 1 because FragData is color space
  gl_FragData[2] = vec4(lightmapCoords, 0.0f, 1.0f);

In composite.vsh and composite.fsh, we simply declare uniform sampler2D colortex2; and sample the lightmap buffer using vec2 lightmap = texture2D(colortex2, tex).rg; and output the lightmap. Note that we should also specify how to read the buffer by adding comment.

const int colortex0Format = RGBA16;
const int colortex1Format = RGBA16;
const int colortex2Format = RGBA16;
Minecraft Lightmap: red channel is object light while green channel is skylight
Minecraft Lightmap: red channel is object light while green channel is skylight

We can write a function to tint the albedo color using lightmap:

vec3 getLightColor(in vec2 lightMap) {
  const vec3 torchColor = vec3(1, 0.43, 0.09);
  const vec3 skyColor = vec3(0.05f, 0.15f, 0.3f);
  // Multiply each part of the light map with it's color
  vec3 torchLighting = lightMap.x * torchColor;
  vec3 skyLighting = lightMap.y * skyColor;
  // Add the lighting togther to get the total contribution of the lightmap the final color.
  return torchLighting + skyLighting;

Implementing Shadow

To implement shadow in a deferred rendering way, we need to know whether a surface can be seen from the light source. So we need a “camera” from the light source. To link the two camera together, we need to reconstruct 3D environment using depth map and then project it onto the “camera” from the light source. Then we ask, whether the depth value of our projection matches the actual depth from the “camera”. If so, then we have a light hit for that specific pixel.

Similar to light map, we can render the depth buffer: uniform sampler2D depthtex0;.

Depth Map
Depth Map

Here is a funny looking image represent the projection of our screen space reconstructed environment onto the “camera” at the sun. The actual projection looks like this:

  vec3 clipSpace = vec3(tex, depth) * 2.0f - 1.0f;
  vec4 viewW = gbufferProjectionInverse * vec4(clipSpace, 1.0f);
  vec3 view = viewW.xyz / viewW.w;
  vec4 world = gbufferModelViewInverse * vec4(view, 1.0f); // move from eye to feet - Minecraft's world space
  vec4 shadowSpace = shadowProjection * shadowModelView * world;

Yes. The inverse of projection matrix. Although it is not invertible, it can still reconstruct some part of the world we see on screen. This is exactly the reason why we save time in deferred rendering.

Shadow Space
Shadow Space

After that, we compare the depth value using step function instead of a branch:

 float shadow = step(sampleCoords.z, unclippedShadow);
Raw Shadow Map
Raw Shadow Map

Notice the shadow has strange z-fighting looking pattern because half of pixel think there is something in between it and the sun due to floating point precision error.

Shadow Map with Floating Point Correction
Shadow Map with Floating Point Correction

We fix it by adding a small constant.

 float shadow = step(sampleCoords.z - 0.001f, unclippedShadow);

But the shadow is rounded. This is due to the low resolution of the shadow map. The resolution is related to screen resolution. Because the sun is very far away and it need to capture the entire world loaded, shadow near us is only represented by a few pixels. Therefore, we need more resolution near the player and less resolution for far away shadows.

For both shadow map creation and sampling, we distort the shadow by stretcing the center.

vec2 distortPosition(in vec2 position) {
  float centerDistance = (abs(position.x) + abs(position.y))/2.0;
  float distortionFactor = mix(1.0f, centerDistance, 0.9f);
  return position / distortionFactor;
Shadow Map with Floating Point Correction and Distortion
Shadow Map with Floating Point Correction and Distortion

Now the shadow looks better. But the shadow still seems too sharp near the edge. We will fix it later. // TODO: future direction

Implementing Gamma Correction and Dynamic Lighting

The world doesn’t look too good because we changed the way we render Minecraft completely. Time to fine tune some color. Here is how default Minecraft looks like.

Default Minecraft Looking
Default Minecraft Looking

This is what we got now. Although it is at noon, the sun light is sharp enough, but since we don’t bounce light, the shadow can never be lit by the sun. We need to simulate non-dirrect lighting by adjusting ambient lighting according to the sun position and correct gamma.

Shaded Minecraft Looking without Gamma Adjustment
Shaded Minecraft Looking without Gamma Adjustment

Some gamma tuning:

float light = sin(float(worldTime)/12000.0f * PI);
float ambient = clamp(light, 0.0f, 0.2f);


vec3 albedo = pow(texture2D(colortex0, tex).rgb, vec3(0.7f));
Default Minecraft Looking After Gamma Adjustment
Default Minecraft Looking After Gamma Adjustment
Shaded Minecraft Looking with Gamma Adjustment
Shaded Minecraft Looking with Gamma Adjustment

Implementing Focus Blur

We first try to implement Gaussian Blur. Although ideally I would use Gaussian distribution, but gaussian distribution is too complex: it joint p.d.f. (because we need x and y) involves multiple computationally heavy powers and divisions and it doesn’t have c.d.f., so I simply use distance to approximate.

vec4 albedos = vec4(0.0f);
  float totalWeight = 0.0f;
  distance(vec2(1.0f), vec2(1.0f));
  for (int i = -BLUR; i <= BLUR; i++) {
    for (int j = -BLUR; j <= BLUR; j++) {
      vec2 shift = texcoords.st + vec2(float(i)/screenShift, float(j)/screenShift);
      float dist = distance(texcoords.st, shift) + 1.0f; // add 1 to avoid division by 0
      totalWeight += dist;
      albedos += texture2D(gcolor, shift) / dist;
  vec4 albedo = albedos / totalWeight;
L2 Blur by Distance
L2 Blur by Distance

However, there are 2 issues:

  1. L2 blur has some transparency issues near edge of renderer.
  2. It jumps too quickly when eye moves fast. I guess it needs to be implemented between frames for smooth gradual blur. The say way you would implement motion blur.


And then we are done for today. There are many, many more improvements and many many ideas I haven’t tried. Some basics are: soft shadow, water reflection, water refraction, bloom. I was researching on Screen Space Reflection (SSR) for water that involves ray marching.



Different Areas of Shader Programming

Ray Marching

Ray Marching Example from http://charstiles.com/raymarch/
Ray Marching Example from http://charstiles.com/raymarch/

Cloud Generation

Clouds by iq on https://www.shadertoy.com/view/XslGRr
Clouds by iq on https://www.shadertoy.com/view/XslGRr

Computational Fluid Dynamics

Spilled by iq on https://www.shadertoy.com/view/MsGSRd
Spilled by iq on https://www.shadertoy.com/view/MsGSRd

Signed Distance Field

Mandelbrot - distance by iq on https://www.shadertoy.com/view/lsX3W4
Mandelbrot – distance by iq on https://www.shadertoy.com/view/lsX3W4

Shader for Games

Minecraft Shaders
Minecraft Shaders

Simple Mincraft Shader
Full Minecraft Shader Tutorial

Other Resources

Shader Tutorial by Shawn
Cheatsheet by Shawn
Online IDE by Shawn

Untitled by 安田現象



`2ch3` is a web3 social media owned by the people: Get access to sublayers of any website to post and read people’s comments on blockchain! This extension is intended to democratize social media owned by tech giants.
1. Download and setup `Metamask`
2. Download and install our plugin by downloading the repository as zip and add as a Chrome extension
3. Enable the extension and long-click on anywhere on the website to start posting messages.

– Post any text on any website permanently, with 1000 messages/dollar
– Edit your message
– View other people’s messages on the same webpage, including date and author

– different width of the screen, text warp, make sure it doesn’t destroy the website
– Challenge to optimize for the same URL
– comment, edit, seamless experience

– reply: for interaction
– listen to the event stream (same as a socket)
– add an optional text filter system on frontend

Download here: https://github.com/KokeCacao/2ch3


Shiftspace's Main Interface
Shiftspace’s Main Interface
Shiftspace's Shifts
Shiftspace’s Shifts
Shiftspace's Network of Shifts
Shiftspace’s Network of Shifts

Shiftspace: A browser extension (by Dan Phiffer and Mushon Zer-Aviv, ShiftSpace) that enables collaboratively annotating, editing and shifting the web.


  • What is it: Shiftspace, like my project, was inspired by the control over how users interact on the Internet. In 2007, the Shiftspace team managed to implement 4 different styles of interaction on the meta-layer created by Shiftspace. Users can put “Notes”, highlight text, replace images, and even edit the source code of a website collaboratively. The project didn’t last till today. In fact, their official demo page and website are not accessible.
  • Good: When I came up with the idea of my project, I had a sense that someone must already made something that enables web-space-specific comments. Here it is: a chrome extension made just for that. Users can develop new plugins for Shiftspace. User can make a “shift” that connect other pages of the website, forming a network of links. The shifts show some relevent information such as the time when something is posted. The idea can be revolutionary, it is worth thinking about why such a promising project failed eventually.
  • Bad: The interface is not very convenient as users had to press SHIFT and SPACE and then click one of the buttons to use it. The big note can cover important contents of the webpage (they should be made transparent when the cursor is near). The image-swap and sourcecode edit is too invasive and will eventually make users confused about whether it is the original content of the website. The extension does not intend to store any information about the author of the comments, which disables making personal connections between website commenters.
  • Inspirations: There are of course many challenges to this idea. The most important one is how can one know the two website links are pointing to the same content. Taking the Youtube link, for example, two distinct videos are stored under a query ?v=xxxxxx of video id, but other queries are about giving Youtube metadata. We would like two notes about the same video, although browsed in slightly different url, to appear together. It is challenging to achieve this. Other problems involve the long-tail distribution of visitors. How do you manage comments on https://www.google.com the main page? How do you balance comments with original content? A seemingly simple extension would take a lot of effort to make.
Shiftspace's Source Editing Interface
Shiftspace’s Source Editing Interface


Play it here: https://kokecacao.me/page/Course/S22/60-212/code/p5/dino/index.html

(Note: No sketches were created in the process of making this project)

Google Dino but play by blinking eyes. The purpose of this remake is to force the player to close their eyes in a game that requires actively looking at the screen. It also requires players with subconscious control over their eyes that are semi-automatic in daily life. By playing this version of Google Dino, players get the opportunity to exercise their eyes during a brief internet disconnection perhaps between long periods of staring at the screens. (Note: the program also has voice input to control dino’s ducking. However, I did not have the opportunity to test it.)
The improved blink detection is dynamically calibrated so that different devices can use it with little performance difference. The graphics are borrowed from a public repository (with slight modifications) to closely resemble the original version of Google Dino.



Zach Lieberman, Más Que La Cara Overview (~12 minute read)
Kyle McDonald, Appropriating New Technologies: Face as Interface (~15 minute read)
Last Week Tonight: Face Recognition (21 minutes)
Joy Buolamwini: How I’m fighting bias in algorithms (9 minutes)
Nabil Hassein, Against Black Inclusion in Facial Recognition (~5 minutes)

Many issues of facial (biometric) recognition are highlighted in the above links including privacy concerns, surveillance concerns, and machine bias… I won’t reiterate them here.

Today, the combination of face detection with publicly available social network information can correctly predict your Facebook profile and the first five digits of your SSN for a third of the public, in under three seconds:
Our study is less about face recognition and more about privacy concerns raised by the convergence of various technologies. There is no obvious answer and solution to the privacy concerns raised by widely available face recognition and identified (or identifiable) facial images. Google’s Eric Schmidt observed that, in the future, young individuals may be entitled to change their names to disown youthful improprieties. It is much harder, however, to change someone’s face.

In reading one of those articles, the above sentences shocked me again although I have known this fact long ago (except for the 5 digits of the SSN part). While reading those articles, two thoughts came to my mind as a scalable way to combat facial recognition software.

  1. There are many adversarial tools for directly modifying the faces. One example is Adversarial Mask. However, those solutions are infeasible as people don’t even want to put on their masks for COVID precaution. We need a better solution.
  2. Facial Filters in Cameras: we can implement algorithms that recognize and modify detected faces slightly in the output image so that while my close friends can correctly identify me but the same task will be hard for computer databases that have millions of faces. This can be achievable because we, as humans, can only encode a small number of facial features in our long-term memory: the size of extracted facial features for the human brain should be way smaller than that for a computer. This way, if all faces can be slightly modified right after taking the picture, the entire Internet would contain fake faces which will make face searcher harder.

Now, think about it. If Facebook (ironically “face”-book, now got a better name), Google, or Apple can implement algorithms like this on their user frontend before any data got transmitted, the majority of images online will become unsearchable.




Using: wikiart_16384 + ViT-B/32 + default paremeter
Prompt: A student suffering from his coding homework

100 iterations
100 iterations
300 iterations
300 iterations

The image quality is not very good as other synthesizers based on image inputs (compared to models related to style
transfer) since the natural language processing pipeline restricted the latent space or because they are not trained
end-to-end. The style of the images generated can get very cliche very soon and the tool doesn’t give the artists
very much control over the generated image.





Using: Narrative Device
Themes: coding, homework
Result: The teacher looked up from his computer screen and saw that his student had not yet arrived. “I’m going to
have to start grading your homework,” he said, “but I want to make sure you are doing it right.”
Comment: This has nothing to do with coding homework. The model interpret coding as teacher’s computer
screen. The result might by highly “plagiarized” from a story book.

Using: inferkit

The teacher looked up from his computer screen and saw that his student had not yet arrived. "I'm going to have to start grading your homework," he said, "but I want to make sure you are doing it right."

The student sighed, rolled his eyes, and said, "There's no point.

This is the ninth grade.

You'll do whatever I say and there's nothing I can do about it. "He sat down, and opened his book.

His teacher put his name in the "Now On and shook his head.

"I get to do this each and every night. Why can't you do it during the day?"

"Well," the teacher said, "when I set my alarm clock for noon, my friend buys me coffee.

I set it back to 5: 30 and I'm still awake.

It's got to be midnight by the time you go to bed.

There's no point in going to sleep."

The student sighed and closed his book.

Comment: The model only seems to pay attention to the previous 2 sentences and major characters. Although attention (to
characters) is all you need, but it is hard for the model to generate a global level theme of the text that is
consistent and relevant.




Genre Picture
Genre Picture
Genre Parameter
Genre Parameter
Breeder Picture
Breeder Picture
Breeding Parameter
Breeding Parameter

The art breeder model is very cool. It magically preserves the lower-level pattern of oil paintings with relatively
high resolution. This quality is often, at least to me, hard to achieve and requires a lot of hyperparameter tuning
and GPU power. The fact that it can run on the browser this fast is amazing (is it running on a browser though? Or
does it use backend GPU with querying?). Also, it is very rare to have a user interface that allows both breeding
and gene-editing. I wonder if “breeding” means parameter breeding rather than pixel-wise breeding.