{"id":2290,"date":"2022-04-27T02:53:38","date_gmt":"2022-04-27T06:53:38","guid":{"rendered":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/?p=2290"},"modified":"2022-04-27T02:53:38","modified_gmt":"2022-04-27T06:53:38","slug":"koke_cacao-finaldocumentation","status":"publish","type":"post","link":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/koke_cacao\/04\/27\/koke_cacao-finaldocumentation\/","title":{"rendered":"Koke_Cacao \u2013 FinalDocumentation"},"content":{"rendered":"<p><a href=\"https:\/\/kokecacao.me\/page\/Post\/Writing_Minecraft_Shader.md\">https:\/\/kokecacao.me\/page\/Post\/Writing_Minecraft_Shader.md<\/a><\/p>\n<p>&nbsp;<\/p>\n<nav>\n<div><a style=\"float: left;\" href=\"OpenGL_Rendering.md\">Prev<\/a><\/div>\n<\/nav>\n<div id=\"markdown-page\" class=\"container markdown-page\">\n<h1 align=\"center\">Writing Minecraft Shader<\/h1>\n<figure><img decoding=\"async\" src=\"https:\/\/kokecacao.me\/page\/Post\/imgs\/2022-04-26_22.50.16.png\" alt=\"Final Image of In-Progress Minecraft Shader\" data-zoom-image=\"\" \/><figcaption>Final Image of In-Progress Minecraft Shader<\/figcaption><\/figure>\n<h2><a href=\"#What_is_Shader?\" name=\"What_is_Shader?\">What is Shader?<\/a><\/h2>\n<h3><a href=\"#Use_Case\" name=\"Use_Case\">Use Case<\/a><\/h3>\n<p>\/\/ TODO: case study<\/p>\n<h3><a href=\"#Terminology\" name=\"Terminology\">Terminology<\/a><\/h3>\n<p>Shader, roughly speaking are functions that transform geometry to pixel on screen. There are many things people called shader. For example, in Unity, shader is defined:<\/p>\n<p><code>.shader<\/code> file<\/p>\n<ul>\n<li><code>properties<\/code>: customizable user input (texture, normal, &#8230;)<\/li>\n<li><code>hidden properties<\/code>: transformation matrix, mesh<\/li>\n<li><code>shader<\/code>: a combination of shaders for different platform\n<ul>\n<li><code>subshader #1<\/code>: high quality render use this\n<ul>\n<li><code>.vsh<\/code><\/li>\n<li><code>.fsh<\/code><\/li>\n<li>&#8230;<\/li>\n<\/ul>\n<\/li>\n<li><code>subshader #2<\/code>: low quality render use this\n<ul>\n<li>&#8230;<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<blockquote><p>This part is inspired by <a href=\"https:\/\/www.youtube.com\/watch?v=kfM-yu0iQBk\">Freya Holm\u00e9r&#8217;s Video<\/a>.<\/p><\/blockquote>\n<p>For consistency, when the word <code>shader<\/code> is mentioned, we refer a combination of files (that we called shader programs) that looks like <code>.vsh<\/code> and <code>.fsh<\/code>.<\/p>\n<h3><a href=\"#General_Shader_Pipeline_with_Vertex_and_Fragment_Shader\" name=\"General_Shader_Pipeline_with_Vertex_and_Fragment_Shader\">General Shader Pipeline with Vertex and Fragment Shader<\/a><\/h3>\n<pre><code class=\"language-python\">data = vertices_of_geometry_in_world\r\ndata = data.map(vsh)\r\ndata = rasterization(data)\r\ndata = data.map(fsh) # actually this is in the process of rasterization as depth test is after fsh\r\npixels = antialiasing(data)\r\n\r\n# This is a vertex buffer file written in python style pseudo code\r\ndef vsh(aPos, aNormal, aColor, aUVCoord, ... you define):\r\n  # THIS IS WHAT WE CALLED `VBUFFER`\r\n  # layout (location = 0) in vec3 aPos;\r\n  # layout (location = 1) in vec3 aNormal;\r\n  # layout (location = 2) in vec3 aColor;\r\n\r\n  # THIS IS TRANSFORMATION MATRIX\r\n  # uniform mat4 model;\r\n  # uniform mat4 view;\r\n  # uniform mat4 proj;\r\n  global model\r\n  global view\r\n  global proj\r\n\r\n  # DO MATH HERE: move vertex position, add attributes to vertex\r\n\r\n  # out vec3 crntPos;\r\n  # out vec3 normal;\r\n  # out vec3 color;\r\n  # Notice [gl_Position] here is not specified. We must use this name to tell OpenGL that this is the vertex position, not attributes we defined\r\n  # crntPos is passed for light calculation\r\n  return gl_Position, crntPos, normal, color, ... you define\r\n\r\ndef rasterization(all_vertice):\r\n  # magic interpolation of all other attributes based on [all_vertice[gl_Position]]\r\n  return fragments_and_interpolated_attributes_from_vertices\r\n\r\ndef fsh(crntPos, normal, color):\r\n  # NOTE THAT THESE INPUTS ARE INTERPOLATED VERSION OF OUTPUTS FROM VERTEX SHADER\r\n  # in vec3 crntPos;\r\n  # in vec3 normal;\r\n  # in vec3 color;\r\n\r\n  # uniform vec4 lightColor;\r\n  # uniform vec3 lightPos;\r\n  # uniform vec3 cameraPos;\r\n  global lightColor; # assuming there is one light\r\n  global lightPos; # assuming there is one light\r\n  global cameraPos;\r\n\r\n  # DO MATH HERE: light, shadow, special effect\r\n\r\n  # out vec4 FragColor;\r\n  return FragColor\r\n<\/code><\/pre>\n<blockquote><p>The concept of fragment is nearly identical to pixel, except multiple fragments can contribute to one pixel. This is so that we can implement multisampling or antialiasing. If no multisampling nor antialiasing, then there is 1-to-1 correspondence between fragment and pixel on screen.<\/p><\/blockquote>\n<h3><a href=\"#Shader_Pipeline_with_Vertex,_Geometry,_and_Fragment_Shader\" name=\"Shader_Pipeline_with_Vertex,_Geometry,_and_Fragment_Shader\">Shader Pipeline with Vertex, Geometry, and Fragment Shader<\/a><\/h3>\n<p>The idea is basically the same except we add a geometry shader between vertex and fragment shader.<\/p>\n<pre><code class=\"language-python\">data = vertices_of_geometry_in_world\r\ndata = data.map(vsh)\r\ndata = assemble_to_triangles(data) # returns a list of triangles, lines, or points\r\ndata = data.map(gsh)\r\ndata = rasterization(data)\r\ndata = data.map(fsh) # actually this is in the process of rasterization as depth test is after fsh\r\npixels = antialiasing(data)\r\n\r\ndef vsh(...):\r\n  # Everything is the same except we do not do transform and projection any more. They are done in gsh now.\r\n  pass;\r\n\r\ndef gsh(crntPos, normal, color):\r\n  # layout (points) in;\r\n  # layout (triangle_strip, max_vertices=8) out;\r\n  setInputLayout(points); # Input can only be: points (one point), lines, triangles, lines_adjacency, triangles_adjacency\r\n  setOutputLayout(triangle_strip, max_vertices=8); # Output can only be: points (multiple points), line_strip (multiple lines), triangle_strip (multiple triangles). We want to output vertices at maximum.\r\n\r\n\r\n  # NOTE THAT THESE INPUTS ALL ARRAYS NOW CONTAINING 1, 2, OR 3 ELEMENTS\r\n  # in vec3 crntPos[];\r\n  # in vec3 normal[];\r\n  # in vec3 color[];\r\n\r\n  # some uniform\r\n\r\n  # DO MATH HERE: remove triangle, add mesh to point, add more triangles near this triangle, add or remove attributes to vertex\r\n  vec4 vertexPos = gl_in[0].gl_Position; # for example, get the position of first vertex\r\n\r\n  # Now we create a rectangle consists of 4 vertice\r\n  # Remember to transform and project those vertices\r\n  gl_Position = vec4(...);\r\n  EmitVertex();\r\n  gl_Position = vec4(...);\r\n  EmitVertex();\r\n  gl_Position = vec4(...);\r\n  EmitVertex();\r\n  gl_Position = vec4(...);\r\n  EmitVertex();\r\n  EndPrimitives();\r\n\r\n  # We also want to create a line\r\n  gl_Position = vec4(...);\r\n  EmitVertex();\r\n  gl_Position = vec4(...);\r\n  EmitVertex();\r\n  EndPrimitives();\r\n\r\n  # out vec3 cat;\r\n  return cat\r\n\r\ndef fsh(cat):\r\n  # in vec3 cat;\r\n  pass\r\n<\/code><\/pre>\n<blockquote><p>This section is inspired by <a href=\"https:\/\/www.youtube.com\/watch?v=C8FK9Xn1gUM\">ThinMatrix&#8217;s OpenGL Tutorial 49: Geometry Shader Introduction<\/a>.<\/p><\/blockquote>\n<h2><a href=\"#About_Minecraft_and_Minecraft_Rendering\" name=\"About_Minecraft_and_Minecraft_Rendering\">About Minecraft and Minecraft Rendering<\/a><\/h2>\n<blockquote><p>Minecraft is a popular sandbox survival game and I have a long relationship with this game. Funny enough, my first line of code in a general purpose language environment is a <code>hello world<\/code> printed in the Minecraft server console, and I have since then built a commercial Minecraft (with my costom GTA mode) server hosted about 300k players. To me and most of Minecraft enthusiasms, Minecraft is more than a game as it hosts communities of different interests: creative map makers, adventure map makers, minigame designers, traditional survival players, redstone technicians, youtubers, pvp players, pve players, community builders.<\/p><\/blockquote>\n<p>Now, Minecraft shaders is a program to change how the traditional Minecraft world looks like by taking over the shader engine that is used to render Minecraft. It is a great way to practice <code>glsl<\/code> skill and learn computer graphics because:<\/p>\n<ol>\n<li>Minecraft is exciting. It encourage you to code.<\/li>\n<li>You don&#8217;t have to play with testing geometry, which is boring. Minecraft provides you with a full game you can test on. It lets you build things gradually from easy to hard. You will be challenged with: terrain, lighting, shadow, transparent block, cloud, changing of the sun, moon phases, animals, water, reflective armor, rain, thunder storm, beacon special effect, posion special effect, different biomes&#8230;<\/li>\n<li>Minecraft is beyond toy environment. It provides you with an overview of how actual game rendering is pipelined. It has multiple stages (more than <code>.vsh<\/code>, <code>.fsh<\/code>), use differed rendering, and dynamic load of buffers.<\/li>\n<li>There are existing communities for Minecraft Shader development. <a href=\"https:\/\/wiki.shaderlabs.org\/wiki\/Main_Page\">Join Shader Labs<\/a><\/li>\n<li>You don&#8217;t need to worry about getting attributes from Geometry. <code>OptiFine<\/code>, a mod that optimizes Minecraft renders provides with you many attributes you can use for free. You can find <a href=\"https:\/\/github.com\/sp614x\/OptiFine\/blob\/master\/OptiFineDoc\/doc\/shaders.txt\">OptiFine Documentation Here<\/a>.<\/li>\n<\/ol>\n<p>However, there is one downside of learning shader using Minecraft: the <code>OpenGL<\/code> language version is quiet old. Quote from Shader Lab: <em>&#8220;Anything that says #version 120 will be best. Minecraft 1.17 will use #version 150, but you are not restricted to just these specific #version&#8217;s. You can use any #version that your GPU\/drivers support.&#8221;<\/em><\/p>\n<blockquote><p><code>OpenGL<\/code> is a graphics engine that provides basic structure of rendering pipeline. Its main job is to do basic geometry load and transform, compile shader, rasterization, and talk with GPU.<\/p><\/blockquote>\n<p>Since Minecraft&#8217;s rendering pipeline is way more complex than a simple shader toy example, we will use some specialized terminology:<\/p>\n<ul>\n<li><code>stage<\/code>: there are 4 possible <code>stages<\/code> for each program: <code>.vsh<\/code>, <code>.gsh<\/code>, <code>.fsh<\/code>, <code>.csh<\/code>.<\/li>\n<li><code>shader program<\/code>: refer to <code>shadows, gbuffers, composites, deferred, ...<\/code> one each contains a collection of stages (<code>.vsh<\/code>, <code>.gsh<\/code>, <code>.fsh<\/code>, <code>.csh<\/code>)<\/li>\n<li><code>pipeline<\/code>: a collection of <code>shader programs<\/code><\/li>\n<li><code>pass<\/code>: general term refer to compute from something to every pixel, filling the entire screen space without leaving out any pixel blank (e.g. without distinguish between entities and blocks)<\/li>\n<\/ul>\n<h2><a href=\"#Rendering_Pipeline:_Deferred_Rendering,_Pipelines,_Stages,_and_Buffer\" name=\"Rendering_Pipeline:_Deferred_Rendering,_Pipelines,_Stages,_and_Buffer\">Rendering Pipeline: Deferred Rendering, Pipelines, Stages, and Buffer<\/a><\/h2>\n<h3><a href=\"#Deferred_Rendering\" name=\"Deferred_Rendering\">Deferred Rendering<\/a><\/h3>\n<h4><a href=\"#Naive_Explanation\" name=\"Naive_Explanation\">Naive Explanation<\/a><\/h4>\n<p>\/\/ TODO: remove naive explanation<\/p>\n<p>Forward Rendering: for visible fragment (regardless overlaps), calculate light. We do calculation for every point of geometry surface in the pyramid volume (clip space) before projection to the screen.<\/p>\n<pre><code class=\"language-python\">for (fragment in models):\r\n  for (lightbulb in lights):\r\n    color += calculateLight(fragment, lightbulb)\r\n<\/code><\/pre>\n<figure><img decoding=\"async\" src=\"https:\/\/cdn.tutsplus.com\/cdn-cgi\/image\/width=600\/gamedev\/uploads\/2013\/10\/buffers.png\" alt=\"Color, Depth, and Normal buffers, each of size \u0002klzzwxh:0008\u0003 (Images by astrofa, via Wikimedia Commons.)\" data-zoom-image=\"\" \/><figcaption>Color, Depth, and Normal buffers, each of size <span class=\"arithmatex\"><span class=\"arithmatex\"> W\u00d7H\u00d73<\/span><\/span>&nbsp;<\/p>\n<p>(Images by astrofa, via Wikimedia Commons.)<\/figcaption><\/figure>\n<figure><img decoding=\"async\" src=\"https:\/\/cdn.tutsplus.com\/cdn-cgi\/image\/width=480\/gamedev\/uploads\/2013\/10\/Deferred_rendering_pass_res.jpg\" alt=\"Final lighting (shading) result generated using the three buffers. (Image by astrofa, via Wikimedia Commons.)\" data-zoom-image=\"\" \/><figcaption>Final lighting (shading) result generated using the three buffers. (Image by astrofa, via Wikimedia Commons.)<\/figcaption><\/figure>\n<p>Deferred Rendering: We don&#8217;t need to calculate light for unseen part of the surface. However, as a trade off, we need to reconstruct the 3D world by inverse projection from screen space and calculate the light for every reconstructed fragment.<\/p>\n<pre><code class=\"language-python\">for (fragment in models):\r\n  albedo = get_albedo(fragment)\r\n  normal = get_normal(fragment)\r\n  depth = get_depth(fragment)\r\nfor (pixel in screen):\r\n  for (lightbulb in lights):\r\n    color += calculateLight(pixel, lightbulb, albedo, normal, depth)\r\n<\/code><\/pre>\n<h4><a href=\"#Complex_Explanation\" name=\"Complex_Explanation\">Complex Explanation<\/a><\/h4>\n<p>The idea is that we don&#8217;t want to calculate lights for surfaces that is not visible to the camera. However, traditionally, depth test is done after fragment shader (for pixel, replacing color if smaller depth), and therefore fragment shader is run for every surface inside clip space.<\/p>\n<blockquote><p>Depth testing is done in screen space after the fragment shader has run. Today most GPUs support a hardware feature called early depth testing. Early depth testing allows the depth test to run before the fragment shader runs. (<a href=\"https:\/\/learnopengl.com\/Advanced-OpenGL\/Depth-testing\">Depth testing<\/a>)<\/p><\/blockquote>\n<p>However, as <a href=\"https:\/\/stackoverflow.com\/questions\/56199983\/does-a-fragment-shader-only-run-for-visible-fragments\">Nicol Bolas indicated in Stackoverflow<\/a>, early depth test will not guarantee non-visible fragments to be passed to fragment shader.<\/p>\n<p>Therefore, we need to do 2 passes: first render depth to rasterized fragments, and then assemble to pixels.<\/p>\n<pre><code class=\"language-c\">while(...) { \/\/ render loop\r\n  \/\/ 1. geometry pass: render all geometric\/color data to g-buffer\r\n  glBindFramebuffer(GL_FRAMEBUFFER, gBuffer);\r\n  glClearColor(0.0, 0.0, 0.0, 1.0); \/\/ keep it black so it doesn't leak into g-buffer\r\n  glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);\r\n  gBufferShader.use();\r\n  for(Object obj : Objects) {\r\n      ConfigureShaderTransformsAndUniforms();\r\n      obj.Draw();\r\n  }\r\n  \/\/ 2. lighting pass: use g-buffer to calculate the scene's lighting\r\n  glBindFramebuffer(GL_FRAMEBUFFER, 0);\r\n  lightingPassShader.use();\r\n  BindAllGBufferTextures();\r\n  SetLightingUniforms();\r\n  RenderQuad();\r\n}\r\n<\/code><\/pre>\n<blockquote><p>Pseudocode borrowed from <a href=\"https:\/\/learnopengl.com\/Advanced-Lighting\/Deferred-Shading\">Deferred Shading<\/a>.<\/p><\/blockquote>\n<h3><a href=\"#Pipelines,_Stage_and_Buffer\" name=\"Pipelines,_Stage_and_Buffer\">Pipelines, Stage and Buffer<\/a><\/h3>\n<h4><a href=\"#Stages\" name=\"Stages\">Stages<\/a><\/h4>\n<p>Stages are piece of code in a file that takes in either geometry, fragment, (and <code>uniform<\/code>, <code>buffer<\/code>), output things required by other stages. It is often run on every vertex or fragment. In a typicall simple shader, vertex shader (<code>.vsh<\/code>) and fragment shader (<code>.fsh<\/code>) are two stages. But in Minecraft, there are more stages.<\/p>\n<p>In OptiFine render, we have many many programs each have at maximum 4 stages.<\/p>\n<p>Four Stages in OptiFine:<\/p>\n<ul>\n<li>vertex stage (<code>.vsh<\/code>):\n<ul>\n<li>run for: every vertex<\/li>\n<li>input: vertex attributes (position, color, light level)<\/li>\n<li>output: projected vertex position and attributes if geometry stage does not exists \/\/ QUESTION: what if exist<\/li>\n<\/ul>\n<\/li>\n<li>geometry stage (<code>.gsh<\/code>): Optional \/\/ TODO: experiment with it<\/li>\n<li>fragment stage (<code>.fsh<\/code>): \/\/ TODO: documentation says it is after rasterization. Really?<\/li>\n<li>compute stage (<code>csh<\/code>): Optional. It does not know about any geometry in the world, but can write directly to a buffer at any location. (Normally when a fragment stage writes to a buffer, it is restricted to only writing at the location of the pixel it was assigned to.) \/\/ TODO: try it<\/li>\n<\/ul>\n<blockquote><p>We transfer interpolated data by using <code>varying<\/code>.<\/p><\/blockquote>\n<h4><a href=\"#Pipelines\" name=\"Pipelines\">Pipelines<\/a><\/h4>\n<p>Programs in Pipelines:<\/p>\n<ul>\n<li>Shadows (shadow.fsh\/vsh): suppose to project the world to the sun (a camera looking at player from the sun). We need its depth map (z-buffer) for calculating shadow.<\/li>\n<li>Gbuffers (files starting with gbuffers_): render terrain, entities, sky (later than <code>shadow<\/code>, in order or <code>Skybasic<\/code> -&gt; <code>skytextured<\/code> -&gt; <code>terrain<\/code> (opaque blocks, wind effect) -&gt; <code>tile entities<\/code> (entities and entity blocks) -&gt; <code>textured, textured_lit<\/code> (particles) -&gt; <code>deferred<\/code> -&gt; <code>weather<\/code>)<\/li>\n<li>Composites (composite(N) or final): run after all geometry (all the gbuffers). For post-processing effects: lighting, ambient, occlusion, fancy clouds, reflections, refractions&#8230; You can write to as many buffer(s) as you want, with whatever data you want.<\/li>\n<li>Deferred: (deferred(N).fsh\/vsh): similar to the composite programs, but runs in the middle of terrain rendering instead of after it. More specifically, they run after all opaque objects have rendered, and before any transparent objects. There is no real goal here. You can write to as many buffer(s) as you want, with whatever data you want.<\/li>\n<\/ul>\n<table>\n<thead>\n<tr>\n<th align=\"center\">Program Name<\/th>\n<th align=\"center\">Rendered Geometry<\/th>\n<th align=\"center\">Fallback<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td align=\"center\"><strong>Shadow Map<\/strong><\/td>\n<td align=\"center\"><\/td>\n<td align=\"center\"><\/td>\n<\/tr>\n<tr>\n<td align=\"center\">shadow<\/td>\n<td align=\"center\">all<\/td>\n<td align=\"center\">none<\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><strong>Shadow Composite<\/strong><\/td>\n<td align=\"center\"><\/td>\n<td align=\"center\"><\/td>\n<\/tr>\n<tr>\n<td align=\"center\">shadowcompshadowcomp&lt;1-15&gt;<\/td>\n<td align=\"center\"><\/td>\n<td align=\"center\">none<\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><strong>Prepare<\/strong><\/td>\n<td align=\"center\"><\/td>\n<td align=\"center\"><\/td>\n<\/tr>\n<tr>\n<td align=\"center\">prepareprepare&lt;1-15&gt;<\/td>\n<td align=\"center\"><\/td>\n<td align=\"center\">none<\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><strong>G-Buffers<\/strong><\/td>\n<td align=\"center\"><\/td>\n<td align=\"center\"><\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_basic<\/td>\n<td align=\"center\">leash<\/td>\n<td align=\"center\">none<\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_line<\/td>\n<td align=\"center\">block selection, fishing line<\/td>\n<td align=\"center\">gbuffers_basic<\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_textured<\/td>\n<td align=\"center\">particles<\/td>\n<td align=\"center\">gbuffers_basic<\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_textured_lit<\/td>\n<td align=\"center\">lit\/emissive particles, world border<\/td>\n<td align=\"center\">gbuffers_textured<\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_skybasic<\/td>\n<td align=\"center\">sky, horizon, stars, void<\/td>\n<td align=\"center\">gbuffers_basic<\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_skytextured<\/td>\n<td align=\"center\">sun, moon<\/td>\n<td align=\"center\">gbuffers_textured<\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_clouds<\/td>\n<td align=\"center\">clouds<\/td>\n<td align=\"center\">gbuffers_textured<\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_terrain<\/td>\n<td align=\"center\">opaque geometry (including cutout transparency)<\/td>\n<td align=\"center\">gbuffers_textured_lit<\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_damagedblock<\/td>\n<td align=\"center\">damaged block overlay<\/td>\n<td align=\"center\">gbuffers_terrain<\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_block<\/td>\n<td align=\"center\">block\/tile entities<\/td>\n<td align=\"center\">gbuffers_terrain<\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_beaconbeam<\/td>\n<td align=\"center\">beacon beam<\/td>\n<td align=\"center\">gbuffers_textured<\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_entities<\/td>\n<td align=\"center\">entities<\/td>\n<td align=\"center\">gbuffers_textured_lit<\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_entities_glowing<\/td>\n<td align=\"center\">glowing entities (spectral effect)<\/td>\n<td align=\"center\">gbuffers_entities<\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_armor_glint<\/td>\n<td align=\"center\">armor glint overlay<\/td>\n<td align=\"center\">gbuffers_textured<\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_spidereyes<\/td>\n<td align=\"center\">eyes of spiders, endermen and enderdragons<\/td>\n<td align=\"center\">gbuffers_textured<\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_hand<\/td>\n<td align=\"center\">hand, opaque handheld items<\/td>\n<td align=\"center\">gbuffers_textured_lit<\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_weather<\/td>\n<td align=\"center\">rain, snow<\/td>\n<td align=\"center\">gbuffers_textured_lit<\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><strong>Deferred<\/strong><\/td>\n<td align=\"center\"><\/td>\n<td align=\"center\"><\/td>\n<\/tr>\n<tr>\n<td align=\"center\">deferreddeferred&lt;1-15&gt;<\/td>\n<td align=\"center\"><\/td>\n<td align=\"center\">none<\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><strong>Translucent G-Buffers<\/strong><\/td>\n<td align=\"center\"><\/td>\n<td align=\"center\"><\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_water<\/td>\n<td align=\"center\">translucent geometry<\/td>\n<td align=\"center\">gbuffers_terrain<\/td>\n<\/tr>\n<tr>\n<td align=\"center\">gbuffers_hand_water<\/td>\n<td align=\"center\">translucent handheld items<\/td>\n<td align=\"center\">gbuffers_hand<\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><strong>Composite<\/strong><\/td>\n<td align=\"center\"><\/td>\n<td align=\"center\"><\/td>\n<\/tr>\n<tr>\n<td align=\"center\">compositecomposite&lt;1-15&gt;<\/td>\n<td align=\"center\"><\/td>\n<td align=\"center\">none<\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><strong>Final<\/strong><\/td>\n<td align=\"center\"><\/td>\n<td align=\"center\"><\/td>\n<\/tr>\n<tr>\n<td align=\"center\">final<\/td>\n<td align=\"center\"><\/td>\n<td align=\"center\">none<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h4><a href=\"#Buffer\" name=\"Buffer\">Buffer<\/a><\/h4>\n<p>Buffer (Framebuffer Attachments) are memory shared accross different <code>program<\/code> (with special permission: some earlier stage cannot access buffers only created for later stage). This is especially useful for deferred rendering. For example, any program executed after <code>shadow<\/code> program can then access depth buffer created by <code>shadow<\/code> program.<\/p>\n<blockquote><p>Quote from <a href=\"https:\/\/pastebin.com\/aB5MJ7aN\">Shader concepts by BUILDERB0Y<\/a>: Create 2 buffers. one is a material buffer, the other is the translucent buffer. Make all transparent objects output their color to the translucent buffer, and a number representing their ID (passed in with varyings) to the material buffer. Composite can read the material buffer, and mix the translucent buffer with the opaque color buffer differently depending on the ID. This will allow effects such as fog behind water, or only applying reflections to glass and water but not slime blocks or nether portals. As you may have guessed though, the material buffer can only store one ID per pixel. In most cases, the ID will be that of the closest transparent object to the camera. Everything behind it will be ignored. This means that if you look at water through stained glass, suddenly it won&#8217;t have thick blue fog anymore. A lot of shader packs have similar issues. Sadly, there&#8217;s no easy way to fix this. Still, this should give you an idea of what is and isn&#8217;t possible to do with OptiFine&#8217;s pipeline.<\/p><\/blockquote>\n<h2><a href=\"#Implementing_The_Shader\" name=\"Implementing_The_Shader\">Implementing The Shader<\/a><\/h2>\n<figure><img decoding=\"async\" src=\"https:\/\/kokecacao.me\/page\/Post\/imgs\/2022-04-26_22.45.47.png\" alt=\"Default Minecraft Looking\" data-zoom-image=\"\" \/><figcaption>Default Minecraft Looking<\/figcaption><\/figure>\n<h3><a href=\"#Understand_Deferred_Rendering\" name=\"Understand_Deferred_Rendering\">Understand Deferred Rendering<\/a><\/h3>\n<p>So how to write Minecraft shader? Minecraft already has its shader. Instead of adding, we replace them. We first play with final pass <code>final.vsh<\/code> and <code>final.fsh<\/code>.<\/p>\n<p>We play with vertex shader first.<\/p>\n<pre><code class=\"language-c\">#version 140\r\nvarying vec4 texcoord;\r\nvoid main() {\r\n  gl_Position = gl_ModelViewProjectionMatrix * (gl_Vertex + vec4(vec2(0.1f, 0.25f), 0.0f, 0.0f));\r\n  texcoord = gl_MultiTexCoord0; \/\/ vec4 of texture output\r\n}\r\n<\/code><\/pre>\n<figure><img decoding=\"async\" src=\"https:\/\/kokecacao.me\/page\/Post\/imgs\/2022-04-26_22.33.30.png\" alt=\"Shifting Screen Vertex\" data-zoom-image=\"\" \/><figcaption>Shifting Screen Vertex<\/figcaption><\/figure>\n<p>We notice that the entire screen is shifted. This is a feature of deferred rendering. The vertex shader does not take in actual vertice, but rather takes a simple plane.<\/p>\n<pre><code class=\"language-c\">#version 140\r\nvarying vec4 texcoord; \/\/ screen coord from 0 to 1\r\nuniform sampler2D gcolor; \/\/ texture storing screen color\r\nvoid main() {\r\n  vec4 albedo = texture2D(gcolor, texcoord.st);\r\n  gl_FragColor = albedo;\r\n}\r\n<\/code><\/pre>\n<p>In fragment shader, we simply pass albedo from the buffer to the next buffer <code>gl_FragColor<\/code>. Since it is <code>final<\/code>, the screen will directly output this color as pixel color.<\/p>\n<h3><a href=\"#Implement_Ambient_Light_and_Color\" name=\"Implement_Ambient_Light_and_Color\">Implement Ambient Light and Color<\/a><\/h3>\n<p>Time to implement light. We need to get the lightmap (as well as normals, colors, and texture coordinate) from vertex shader <code>gbuffers_terrain.vsh<\/code> and pass into <code>gbuffers_terrain.fsh<\/code><\/p>\n<pre><code class=\"language-c\">varying vec4 texcoord;\r\nvarying vec3 normal;\r\nvarying vec4 color;\r\nvarying vec2 lightmapCoords;\r\nvoid main() {\r\n  gl_Position = ftransform();\r\n  texcoord = gl_MultiTexCoord0;\r\n  normal = gl_NormalMatrix * gl_Normal;\r\n  color = gl_Color;\r\n  lightmapCoords = (lightmapCoords * 33.05f \/ 32.0f) - (1.05f \/ 32.0f);\r\n}\r\n<\/code><\/pre>\n<p>In fragment shader <code>gbuffers_terrain.fsh<\/code>, we simply store them in gbuffer for deferred rendering. The comment <code>\/* RENDERTARGETS: 0,1,2 *\/<\/code> tells OptiFine that buffer position in our code <code>0, 1, 2<\/code> correspond to actual buffer position <code>0, 1, 2<\/code>. This is so that OptiFine does not need to attach other buffers that we don&#8217;t use.<\/p>\n<pre><code class=\"language-c\">varying vec4 texcoord;\r\nvarying vec3 normal;\r\nvarying vec4 color; \/\/ for biome specific color and ambient occlusion\r\nvarying vec2 lightmapCoords;\r\n\r\nuniform sampler2D texture;\r\n\r\nvoid main() {\r\n  vec4 albedo = texture2D(texture, texcoord.st) * color;\r\n  \/* RENDERTARGETS: 0,1,2 *\/\r\n  gl_FragData[0] = albedo;\r\n  gl_FragData[1] = vec4(normal * 0.5f + 0.5f, 1.0f); \/\/ normal is -1 to 1. We need to fit in 0 to 1 because FragData is color space\r\n  gl_FragData[2] = vec4(lightmapCoords, 0.0f, 1.0f);\r\n}\r\n<\/code><\/pre>\n<p>In <code>composite.vsh<\/code> and <code>composite.fsh<\/code>, we simply declare <code>uniform sampler2D colortex2;<\/code> and sample the lightmap buffer using <code>vec2 lightmap = texture2D(colortex2, tex).rg;<\/code> and output the lightmap. Note that we should also specify how to read the buffer by adding comment.<\/p>\n<pre><code class=\"language-c\">\/*\r\nconst int colortex0Format = RGBA16;\r\nconst int colortex1Format = RGBA16;\r\nconst int colortex2Format = RGBA16;\r\n*\/\r\n<\/code><\/pre>\n<figure><img decoding=\"async\" src=\"https:\/\/kokecacao.me\/page\/Post\/imgs\/2022-04-26_22.40.48.png\" alt=\"Minecraft Lightmap: red channel is object light while green channel is skylight\" data-zoom-image=\"\" \/><figcaption>Minecraft Lightmap: red channel is object light while green channel is skylight<\/figcaption><\/figure>\n<p>We can write a function to tint the albedo color using lightmap:<\/p>\n<pre><code class=\"language-c\">vec3 getLightColor(in vec2 lightMap) {\r\n  const vec3 torchColor = vec3(1, 0.43, 0.09);\r\n  const vec3 skyColor = vec3(0.05f, 0.15f, 0.3f);\r\n  \/\/ Multiply each part of the light map with it's color\r\n  vec3 torchLighting = lightMap.x * torchColor;\r\n  vec3 skyLighting = lightMap.y * skyColor;\r\n  \/\/ Add the lighting togther to get the total contribution of the lightmap the final color.\r\n  return torchLighting + skyLighting;\r\n}\r\n<\/code><\/pre>\n<h3><a href=\"#Implementing_Shadow\" name=\"Implementing_Shadow\">Implementing Shadow<\/a><\/h3>\n<p>To implement shadow in a deferred rendering way, we need to know whether a surface can be seen from the light source. So we need a &#8220;camera&#8221; from the light source. To link the two camera together, we need to reconstruct 3D environment using depth map and then project it onto the &#8220;camera&#8221; from the light source. Then we ask, whether the depth value of our projection matches the actual depth from the &#8220;camera&#8221;. If so, then we have a light hit for that specific pixel.<\/p>\n<p>Similar to light map, we can render the depth buffer: <code>uniform sampler2D depthtex0;<\/code>.<\/p>\n<figure><img decoding=\"async\" src=\"https:\/\/kokecacao.me\/page\/Post\/imgs\/2022-04-26_22.41.50.png\" alt=\"Depth Map\" data-zoom-image=\"\" \/><figcaption>Depth Map<\/figcaption><\/figure>\n<p>Here is a funny looking image represent the projection of our screen space reconstructed environment onto the &#8220;camera&#8221; at the sun. The actual projection looks like this:<\/p>\n<pre><code class=\"language-c\">  \/\/ SHADOW STUFF\r\n  vec3 clipSpace = vec3(tex, depth) * 2.0f - 1.0f;\r\n  vec4 viewW = gbufferProjectionInverse * vec4(clipSpace, 1.0f);\r\n  vec3 view = viewW.xyz \/ viewW.w;\r\n  vec4 world = gbufferModelViewInverse * vec4(view, 1.0f); \/\/ move from eye to feet - Minecraft's world space\r\n  vec4 shadowSpace = shadowProjection * shadowModelView * world;\r\n<\/code><\/pre>\n<blockquote><p>Yes. The inverse of projection matrix. Although it is not invertible, it can still reconstruct some part of the world we see on screen. This is exactly the reason why we save time in deferred rendering.<\/p><\/blockquote>\n<figure><img decoding=\"async\" src=\"https:\/\/kokecacao.me\/page\/Post\/imgs\/2022-04-26_22.45.27.png\" alt=\"Shadow Space\" data-zoom-image=\"\" \/><figcaption>Shadow Space<\/figcaption><\/figure>\n<p>After that, we compare the depth value using <code>step<\/code> function instead of a branch:<\/p>\n<pre><code class=\"language-c\"> float shadow = step(sampleCoords.z, unclippedShadow);\r\n<\/code><\/pre>\n<figure><img decoding=\"async\" src=\"https:\/\/kokecacao.me\/page\/Post\/imgs\/2022-04-26_22.43.46.png\" alt=\"Raw Shadow Map\" data-zoom-image=\"\" \/><figcaption>Raw Shadow Map<\/figcaption><\/figure>\n<p>Notice the shadow has strange z-fighting looking pattern because half of pixel think there is something in between it and the sun due to floating point precision error.<\/p>\n<figure><img decoding=\"async\" src=\"https:\/\/kokecacao.me\/page\/Post\/imgs\/2022-04-26_22.44.25.png\" alt=\"Shadow Map with Floating Point Correction\" data-zoom-image=\"\" \/><figcaption>Shadow Map with Floating Point Correction<\/figcaption><\/figure>\n<p>We fix it by adding a small constant.<\/p>\n<pre><code class=\"language-c\"> float shadow = step(sampleCoords.z - 0.001f, unclippedShadow);\r\n<\/code><\/pre>\n<p>But the shadow is rounded. This is due to the low resolution of the shadow map. The resolution is related to screen resolution. Because the sun is very far away and it need to capture the entire world loaded, shadow near us is only represented by a few pixels. Therefore, we need more resolution near the player and less resolution for far away shadows.<\/p>\n<p>For both shadow map creation and sampling, we distort the shadow by stretcing the center.<\/p>\n<pre><code class=\"language-c\">vec2 distortPosition(in vec2 position) {\r\n  float centerDistance = (abs(position.x) + abs(position.y))\/2.0;\r\n  float distortionFactor = mix(1.0f, centerDistance, 0.9f);\r\n  return position \/ distortionFactor;\r\n}\r\n<\/code><\/pre>\n<figure><img decoding=\"async\" src=\"https:\/\/kokecacao.me\/page\/Post\/imgs\/2022-04-26_22.45.07.png\" alt=\"Shadow Map with Floating Point Correction and Distortion\" data-zoom-image=\"\" \/><figcaption>Shadow Map with Floating Point Correction and Distortion<\/figcaption><\/figure>\n<p>Now the shadow looks better. But the shadow still seems too sharp near the edge. We will fix it later. \/\/ TODO: future direction<\/p>\n<h3><a href=\"#Implementing_Gamma_Correction_and_Dynamic_Lighting\" name=\"Implementing_Gamma_Correction_and_Dynamic_Lighting\">Implementing Gamma Correction and Dynamic Lighting<\/a><\/h3>\n<p>The world doesn&#8217;t look too good because we changed the way we render Minecraft completely. Time to fine tune some color. Here is how default Minecraft looks like.<\/p>\n<figure><img decoding=\"async\" src=\"https:\/\/kokecacao.me\/page\/Post\/imgs\/2022-04-26_22.45.47.png\" alt=\"Default Minecraft Looking\" data-zoom-image=\"\" \/><figcaption>Default Minecraft Looking<\/figcaption><\/figure>\n<p>This is what we got now. Although it is at noon, the sun light is sharp enough, but since we don&#8217;t bounce light, the shadow can never be lit by the sun. We need to simulate non-dirrect lighting by adjusting ambient lighting according to the sun position and correct gamma.<\/p>\n<figure><img decoding=\"async\" src=\"https:\/\/kokecacao.me\/page\/Post\/imgs\/2022-04-26_22.46.26.png\" alt=\"Shaded Minecraft Looking without Gamma Adjustment\" data-zoom-image=\"\" \/><figcaption>Shaded Minecraft Looking without Gamma Adjustment<\/figcaption><\/figure>\n<p>Some gamma tuning:<\/p>\n<pre><code class=\"language-c\">float light = sin(float(worldTime)\/12000.0f * PI);\r\nfloat ambient = clamp(light, 0.0f, 0.2f);\r\n\r\n...\r\n\r\nvec3 albedo = pow(texture2D(colortex0, tex).rgb, vec3(0.7f));\r\n<\/code><\/pre>\n<figure><img decoding=\"async\" src=\"https:\/\/kokecacao.me\/page\/Post\/imgs\/2022-04-26_22.46.10.png\" alt=\"Default Minecraft Looking After Gamma Adjustment\" data-zoom-image=\"\" \/><figcaption>Default Minecraft Looking After Gamma Adjustment<\/figcaption><\/figure>\n<figure><img decoding=\"async\" src=\"https:\/\/kokecacao.me\/page\/Post\/imgs\/2022-04-26_22.50.16.png\" alt=\"Shaded Minecraft Looking with Gamma Adjustment\" data-zoom-image=\"\" \/><figcaption>Shaded Minecraft Looking with Gamma Adjustment<\/figcaption><\/figure>\n<h3><a href=\"#Implementing_Focus_Blur\" name=\"Implementing_Focus_Blur\">Implementing Focus Blur<\/a><\/h3>\n<p>We first try to implement Gaussian Blur. Although ideally I would use Gaussian distribution, but gaussian distribution is too complex: it joint p.d.f. (because we need x and y) involves multiple computationally heavy powers and divisions and it doesn&#8217;t have c.d.f., so I simply use distance to approximate.<\/p>\n<pre><code class=\"language-c\">vec4 albedos = vec4(0.0f);\r\n  float totalWeight = 0.0f;\r\n  distance(vec2(1.0f), vec2(1.0f));\r\n  for (int i = -BLUR; i &lt;= BLUR; i++) {\r\n    for (int j = -BLUR; j &lt;= BLUR; j++) {\r\n      vec2 shift = texcoords.st + vec2(float(i)\/screenShift, float(j)\/screenShift);\r\n      float dist = distance(texcoords.st, shift) + 1.0f; \/\/ add 1 to avoid division by 0\r\n      totalWeight += dist;\r\n      albedos += texture2D(gcolor, shift) \/ dist;\r\n    }\r\n  }\r\n  vec4 albedo = albedos \/ totalWeight;\r\n<\/code><\/pre>\n<figure><img decoding=\"async\" src=\"https:\/\/kokecacao.me\/page\/Post\/imgs\/2022-04-27_02.26.02.png\" alt=\"L2 Blur by Distance\" data-zoom-image=\"\" \/><figcaption>L2 Blur by Distance<\/figcaption><\/figure>\n<p>However, there are 2 issues:<\/p>\n<ol>\n<li><code>L2<\/code> blur has some transparency issues near edge of renderer.<\/li>\n<li>It jumps too quickly when eye moves fast. I guess it needs to be implemented between frames for smooth gradual blur. The say way you would implement motion blur.<\/li>\n<\/ol>\n<h2><a href=\"#Done\" name=\"Done\">Done<\/a><\/h2>\n<p>And then we are done for today. There are many, many more improvements and many many ideas I haven&#8217;t tried. Some basics are: soft shadow, water reflection, water refraction, bloom. I was researching on Screen Space Reflection (SSR) for water that involves ray marching.<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>https:\/\/kokecacao.me\/page\/Post\/Writing_Minecraft_Shader.md &nbsp; Prev Writing Minecraft Shader Final Image of In-Progress Minecraft Shader What is Shader? Use Case \/\/ TODO: case study Terminology Shader, roughly speaking are functions that transform geometry to pixel on screen. There are many things people called shader. For example, in Unity, shader is defined: .shader file properties: customizable user input (texture, &hellip; <a href=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/koke_cacao\/04\/27\/koke_cacao-finaldocumentation\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Koke_Cacao \u2013 FinalDocumentation&#8221;<\/span><\/a><\/p>\n","protected":false},"author":10,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[34],"tags":[],"_links":{"self":[{"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/posts\/2290"}],"collection":[{"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/comments?post=2290"}],"version-history":[{"count":1,"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/posts\/2290\/revisions"}],"predecessor-version":[{"id":2295,"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/posts\/2290\/revisions\/2295"}],"wp:attachment":[{"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/media?parent=2290"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/categories?post=2290"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/tags?post=2290"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}