Author: Hunan

Hunan-FinalDocumentation

The previous posts already contains all the documentation I have for the final project so I’ll just copy them over here.

PART 1

A unity walkaround with procedurally generated terrain of (medieval?) Pittsburgh. A simulator of ancient Pittsburgh, if you will.

An example of heightmap data used as terrain information. This information is available for almost the entirety of the US and many parts of the world at: https://heightmap.skydark.pl/

[Looking at point park from a little bit to the west of (what is now) Grandview Overlook]

For reference:

[Looking at Schendly from CMU]

[Random houses on Mt. Washington (south shore of PGH)]

PART 2

I was working on procedural terrain generation for the first half of this project but because I found a really good package for that, there’s very little left for me to do apart from learning how to use that package. Since my goal was to learn unity, I decided to make something that requires more scripting. So I made a 3D in-game builder (the best way to explain it would be the combination of Kerbal Space Program’s user interaction combined with the mechanics of Minecraft.)  I got to explore scripts, material, audio sources, UI, character controller, physics ray casting, box collider, and mesh renderer. The environment is the HDRP sample scene in Unity. It only took 150 lines of code for the builder and another 50 for a very naive character movement controller, which is quite amazing.

 

Hunan-LookingOutwards03

https://experiments.withgoogle.com/imaginary-soundscape

I found this project and thought it was a really cool use of ML with a lot of potentials. In the demo, the model generates sound based on the street view, giving the user a full audio visual experience from the purely visual street map data. I thought it really utilized DL to its strength — finding patterns for massive datasets. With only a reasonably sized training set, it can be generalized to the massive street view data we have. The underlying model (if I understood/guessed correctly) is a CNN (which is frozen) that takes in the image to generate an embedding, and another CNN that is trained but discarded at inference which takes in the sound file to generate an embedding as close to the one generated by the image CNN for the paired image. The embedding for each sound file is then saved and used as a key to retrieve sound file at inference time. With more recent developments in massive language models (transformers,) we’ve seen evidence (Jukebox, Visual transformers, etc.) showing that they are more or less task agnostic. Since the author mentioned that the model sometimes lacks a full semantic understanding of the context but simply match sound files to the objects seen in the image, these multi-modal models are a promising way to further improve this model to account for more complex contexts. It might also open up some possibilities of remixing the sound files instead of simply retrieving existing sound data to further improve the user experience. This might also see some exciting uses in the gaming/simulation industry. Windows’ new flight simulator is already taking advantage of DL to generate a 3D model (mainly buildings) of the entire earth from satellite imagery, it’s only reasonable to assume that some day we’ll need audio generation to go along with the 3D asset generation for the technology to be used in more games.

hunan-FinalPhase4

I was working on procedural terrain generation for the first half of this project but because I found a really good package for that, there’s very little left for me to do apart from learning how to use that package. Since my goal was to learn unity, I decided to make something that requires more scripting. So I made a 3D in-game builder (the best way to explain it would be the combination of Kerbal Space Program’s user interaction combined with the mechanics of Minecraft.)  I got to explore scripts, material, audio sources, UI, character controller, physics ray casting, box collider, and mesh renderer. The environment is the HDRP sample scene in Unity. It only took 150 lines of code for the builder and another 50 for a very naive character movement controller, which is quite amazing.

 

hunan-Finalphase3

A unity walkaround with procedurally generated terrain of (medieval?) Pittsburgh. A simulator of ancient Pittsburgh, if you will.

An example of heightmap data used as terrain information. This information is available for almost the entirety of the US and many parts of the world at: https://heightmap.skydark.pl/

[Looking at point park from a little bit to the west of (what is now) Grandview Overlook]

For reference:

[Looking at Schendly from CMU]

[Random houses on Mt. Washington (south shore of PGH)]

hunan-FinalPhase2

After watching a bunch of Unity tutorials and poking around in the editor a little bit, I’ve decided to learn the engine.

On the Unity side, I’ll either follow a tutorial or just study a sample project to make a first-person walk-around. (First Person Character Controller, Procedural Mesh.) This likely will not include using any other assets, prefabs, or other functions. Just a character controller object that slides along the terrain.

On the python mesh generation side, I’ll be using the open-elevation API to get the elevation data of a given region. I might also try the sentinelsat library to query the satellite map, which I could use for color or some other purposes. I’ll then use pymesh to make a .obj file to use in Unity.

 

hunan-FinalPhase1

I’m thinking about making a transformation pipeline that can turn a set of GIS data into a virtual scene, not too different from procedurally generated scenes in games. This will consist of two parts: a transformation that is only done once to generate an intermediate set of data from the raw dataset and another set of code that can render the intermediate data into an interactive scene. I’ll call then stage 1 and stage 2.

To account for the indeterministic nature of my schedule, I want to plan different steps and possible pathways, which are listed below.

Stage 1:

  1. Using google earth’s GIS API, start with the basic terrain information for a piece of land and sort out all the basic stuff (reading data into python, python libs for mesh processing, output format, etc.) and test out some possibilities for the transformation.
  2. Start to incorporate the RGB info into the elevation info. See what I can do with those new data.
  3. Find some LiDAR datasets if point clouds can give me more options.

Stage 2:

  1. Just take the code I used for SkyScape (the one with 80k ice particles) and modify it to work with the intermediate format instead of random data.
  2. Make it look prettier by using different meshes, movements, and postprocessings that work with the overall concept.
  3. Using something like oimo.js or enable3D, add physics to the scene to allow for more user interactions and variabilities of the scene.
  4. Enhance user interaction by enhancing camera control, polishing the motions, adding extra interaction possibilities, etc.
  5. (If I have substantial time or if Unity is just a better option for this) Learn Unity and implement the above using Unity istead.

I’ll start with modifying the SkyScape code to work with a terrain mesh, which would give me a working product quickly, and go from there.

hunan-facereadings

Against Black Inclusion & How I’m Fighting Bias

The discussion around algorithmic justice (especially with facial recognition) had been at the center of attention for a while now. I’ve thought a lot about it and concluded the following:

  1. I don’t think we’ll be solving the problem by eliminating the technology. Technological advancements have been a constant in human history except for a few cases where mass destruction (biological, chemical, nuclear weapons) or major ethical challenges (human cloning) are concerned. Facial recognition and its application are not qualitatively different from many other biometric systems such as fingerprinting.
  2. Though facial recognition is more powerful, it is not qualitatively disruptive as other technologies such as cloning humans. Thus, the technology itself should not be the subject of concern — it merely reveals and amplifies the existing systemic injustice and oppression. For example, it was mentioned in one of the articles that the US is planning on using facial recognition to trach everyone who is leaving the country. However, the same tracking system (implemented with fingerprinting) has been in the U.S. for aliens and in China for all citizens. With qualitatively different and destructive technologies, it would make more sense to control the technology itself, as its destructive quality is independent of its ownership and usage. Whereas in this case, it would make more sense to direct the fight for justice towards those who are using the weapon (the state, the government, the capital, etc.) than the weapons themselves (the technology.)
  3. I’m by no means downplaying the potential harm of misusing technology as a tool of oppression. As someone who’s lived in China and seen how technology is fast-tracking the blatant violation of basic human rights in Xinjiang and Tibet (and many other places around the world,) I’ve witnessed the destructive power of information technology. This is not new — IBM’s punchcard and computer systems found some of their earliest uses in the Holocaust and greatly improved the efficiency of ethnic cleansing. However, compared to other technologies, information technology does provide hope for more open-sourced, democratic distribution of power. Unlike military and nuclear power, which are held exclusively by the state, we see more individuals and NGOs leveraging the power of information technology to exert influence on a previously unimaginable scale (a recent example would be the involvement of civilian hackers in the Russian invasion of Ukraine; decentralized cryptocurrency is another example.) Though alarmed by its potential harm, I also remain optimistic for the future of technology and democracy.

hunan-VQGAN+CLIP

Since I don’t have Colab Pro, I spent a lot of time trying to get the notebook working on a local runtime so I can get the bigger models working. But many library issues and differences between Windows and Linux made it very annoying. So I resorted to using a smaller model on Colab.

Lemongrass under the pink sky rendered in blender

hunan-TextSynthesis

This is the first time I’ve used GTP-3 and was honestly a little disappointed. Don’t get me wrong — it’s an amazing model, I just didn’t see the improvements over GPT-2 that I was looking forward to (given the massive increase in parameter size over the already massive GPT-2.) Maybe the improvements in text generation are already marginal and it would take something like DALL-E to tell the difference.

I tried putting in the first sentence of a news article generated by GPT-2 for one of my previous projects into GPT-3 as a prompt. The following is what I got.

The family of a Maryland woman who was killed by a stray bullet last year has settled a wrongful death lawsuit brought by the family of Kimberly Ballinger, a 19-year-old who was struck by a stray bullet in March 2018.

News outlets report attorneys for Ballinger’s family confirmed in a news release that the lawsuit against Baltimore County was settled on Aug. 17 for an undisclosed amount.

Sarah Gelsomino was struck in the neck by the bullet that struck Ballinger as the two were in a car. The lawsuit alleged that Ballinger was inadvertently killed by Gelsomino’s boyfriend, Charles Lee Thornton Jr.

Gelsomino survived and filed a lawsuit in June against Thornton. The lawsuit alleged the shooting was “reckless and unreasonable,” and that Thornton acted recklessly, setting into motion Ballinger’s death.

Advertisement

Thornton was charged in Ballinger’s death. It’s unclear if he has a lawyer.

Copyright © 2018 The Associated Press. All rights reserved. This material may not be published, broadcast, written or redistributed.

This is the original article generated by GPT-2:

The family of a Maryland woman who was killed by a stray bullet last year has settled a wrongful death lawsuit brought by the family of Kimberly Ballinger, a 19-year-old who was struck by a stray bullet in March 2018. Ballinger’s family said they were given about four weeks to file their claims, and on Monday the family settled the case. “The family is relieved that this case is closed,” Ballinger’s family attorney, Stephen R. Kaufman, said in a statement. “They have been unable to pursue their claims against the gun manufacturer that manufactured the bullet and that illegally sold it to the gun shop owner.” The Maryland shooting happened at a gun shop in Fayetteville, Maryland, when Ballinger was shopping with her father and a friend. The store owner, Lorne Ahrens, fatally shot Ballinger, who was a student at Towson University, according to the complaint. The gun shop owner’s wife, Tammy Ahrens, was also shot. “The bullet struck Ms. Ballinger squarely in the back of the head,” the complaint said. “Although Ms. Ballinger did not have a bullet wound, the bullet went through her inner right arm and into her right shoulder, severing a major part of her arm and causing her to lose her life.” Ballinger was struck on the left side of her body and her right arm, the complaint said. Ballinger was in critical condition at a hospital for about 24 hours before being transferred to a hospital for treatment of an arm injury, according to the complaint. The bullet was recovered from her body, and the gun shop owner was arrested and charged with first-degree murder, the complaint said. He had been held on $500,000 bond. Ballinger’s family asked that the $1 million awarded to Ahrens be increased to compensate for Ballinger’s loss of life and a $100,000 reward for information leading to his arrest. The gun shop owner and his wife pleaded not guilty to murder charges. The family is asking for unspecified monetary awards, restitution and attorney’s fees. The family is also asking for the gun shop owner to be held accountable for his actions. CNN’s Artemis Moshtaghian contributed to this report.