Skip to content

Commit

Permalink
Moved section from challenges to E2E approach
Browse files Browse the repository at this point in the history
  • Loading branch information
@pwichmann
pwichmann committed Oct 23, 2021
1 parent 3a07f01 commit dcf540c
Show file tree
Hide file tree
Showing 75 changed files with 2,024 additions and 645 deletions.
4 changes: 4 additions & 0 deletions docs/.buildinfo
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Sphinx build info version 1
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
config: 81dc89e2358da3e57a74791eedde63b6
tags: 645f666f9bcd5a90fca523b33c5a78b7
1 change: 0 additions & 1 deletion docs/.nojekyll
View file

This file was deleted.

59 changes: 58 additions & 1 deletion docs/02_background/DL_with_vector_graphics.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -261,6 +261,11 @@ <h1 class="site-logo" id="site-title">Deep Learning with Vector Graphics</h1>
</i>
</label>
<ul>
<li class="toctree-l2">
<a class="reference internal" href="../03_related_work/further_academic_papers/campbell_and_kautz_2014.html">
Campbell and Kautz (2014) | Learning a Manifold of Fonts
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../03_related_work/further_academic_papers/Das_2020.html">
Das et al. (2020) | BézierSketch: A generative model for scalable vector sketches
Expand Down Expand Up @@ -332,10 +337,62 @@ <h1 class="site-logo" id="site-title">Deep Learning with Vector Graphics</h1>
From strokes to SVG
</a>
</li>
<li class="toctree-l1">
<li class="toctree-l1 has-children">
<a class="reference internal" href="../04_data_representation/common_preprocessing.html">
Common SVG preprocessing
</a>
<input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
<label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
<ul>
<li class="toctree-l2">
<a class="reference internal" href="../04_data_representation/preprocessing/first_steps.html">
First steps
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../04_data_representation/preprocessing/deminifying_path_data.html">
“De-minifying” path data
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../04_data_representation/preprocessing/harmonizing_positioning.html">
Harmonizing command positioning logic
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../04_data_representation/preprocessing/decomposition_of_basic_shapes.html">
Decomposition of basic SVG shapes to paths
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../04_data_representation/preprocessing/reducing_path_commands.html">
Reducing the set of different path commands
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../04_data_representation/preprocessing/normalization.html">
Normalization
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../04_data_representation/preprocessing/path_simplification.html">
Path simplification
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../04_data_representation/preprocessing/finishing_touches.html">
Finishing touches
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../04_data_representation/preprocessing/embedding.html">
Embedding
</a>
</li>
</ul>
</li>
</ul>
<p class="caption">
Expand Down
196 changes: 182 additions & 14 deletions docs/02_background/DL_with_vector_graphics/arguing_for_e2e_approach.html
View file

Large diffs are not rendered by default.

155 changes: 61 additions & 94 deletions docs/02_background/DL_with_vector_graphics/challenges_of_vector_graphics.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -261,6 +261,11 @@ <h1 class="site-logo" id="site-title">Deep Learning with Vector Graphics</h1>
</i>
</label>
<ul>
<li class="toctree-l2">
<a class="reference internal" href="../../03_related_work/further_academic_papers/campbell_and_kautz_2014.html">
Campbell and Kautz (2014) | Learning a Manifold of Fonts
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../../03_related_work/further_academic_papers/Das_2020.html">
Das et al. (2020) | BézierSketch: A generative model for scalable vector sketches
Expand Down Expand Up @@ -332,10 +337,62 @@ <h1 class="site-logo" id="site-title">Deep Learning with Vector Graphics</h1>
From strokes to SVG
</a>
</li>
<li class="toctree-l1">
<li class="toctree-l1 has-children">
<a class="reference internal" href="../../04_data_representation/common_preprocessing.html">
Common SVG preprocessing
</a>
<input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
<label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
<ul>
<li class="toctree-l2">
<a class="reference internal" href="../../04_data_representation/preprocessing/first_steps.html">
First steps
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../../04_data_representation/preprocessing/deminifying_path_data.html">
“De-minifying” path data
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../../04_data_representation/preprocessing/harmonizing_positioning.html">
Harmonizing command positioning logic
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../../04_data_representation/preprocessing/decomposition_of_basic_shapes.html">
Decomposition of basic SVG shapes to paths
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../../04_data_representation/preprocessing/reducing_path_commands.html">
Reducing the set of different path commands
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../../04_data_representation/preprocessing/normalization.html">
Normalization
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../../04_data_representation/preprocessing/path_simplification.html">
Path simplification
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../../04_data_representation/preprocessing/finishing_touches.html">
Finishing touches
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../../04_data_representation/preprocessing/embedding.html">
Embedding
</a>
</li>
</ul>
</li>
</ul>
<p class="caption">
Expand Down Expand Up @@ -483,23 +540,6 @@ <h1 class="site-logo" id="site-title">Deep Learning with Vector Graphics</h1>
4. Linear interpolation in case of step changes
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#current-vectorizers-to-convert-png-to-svg-perform-poorly">
5. Current vectorizers (to convert PNG to SVG) perform poorly
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h4 nav-item toc-entry">
<a class="reference internal nav-link" href="#examples">
Examples
</a>
</li>
<li class="toc-h4 nav-item toc-entry">
<a class="reference internal nav-link" href="#summary-of-limitations-of-current-vectorizers">
Summary of limitations of current vectorizers
</a>
</li>
</ul>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
Expand Down Expand Up @@ -527,7 +567,6 @@ <h1>Challenges of Vector Graphics<a class="headerlink" href="#challenges-of-vect
<li><p>Rasterizing is not a trivial function</p></li>
<li><p>Ambiguous SVG source of a PNG</p></li>
<li><p>Linear interpolation in case of step changes</p></li>
<li><p>Current vectorizers (to convert PNG to SVG) perform poorly</p></li>
</ol>
<p>Each key challenge shall be discussed below.</p>
<div class="section" id="key-challenges">
Expand Down Expand Up @@ -576,91 +615,19 @@ <h3>4. Linear interpolation in case of step changes<a class="headerlink" href="#
<p>But in the vector image space, one image consists of <strong>two separate paths</strong>, and the other consists of <strong>just one</strong>.</p>
<div class="figure align-default" id="eye-1">
<a class="reference internal image-reference" href="../../_images/eye_1.png"><img alt="eye" src="../../_images/eye_1.png" style="width: 400px;" /></a>
<p class="caption"><span class="caption-number">Fig. 4 </span><span class="caption-text">Icon of an eye opened in Inkscape; the eye consists of two separate paths</span><a class="headerlink" href="#eye-1" title="Permalink to this image"></a></p>
<p class="caption"><span class="caption-number">Fig. 12 </span><span class="caption-text">Icon of an eye opened in Inkscape; the eye consists of two separate paths</span><a class="headerlink" href="#eye-1" title="Permalink to this image"></a></p>
</div>
<div class="figure align-default" id="eye-2">
<a class="reference internal image-reference" href="../../_images/eye_2.png"><img alt="eye" src="../../_images/eye_2.png" style="width: 400px;" /></a>
<p class="caption"><span class="caption-number">Fig. 5 </span><span class="caption-text">Icon of an eye opened in Inkscape; the eye consists of just one path</span><a class="headerlink" href="#eye-2" title="Permalink to this image"></a></p>
<p class="caption"><span class="caption-number">Fig. 13 </span><span class="caption-text">Icon of an eye opened in Inkscape; the eye consists of just one path</span><a class="headerlink" href="#eye-2" title="Permalink to this image"></a></p>
</div>
<p>Now, how would linear interpolations between these SVG representations work?
How would a linear interpolation look if one the icons even had the white part of the eye explicity modelled as a shape?
Is this still possible in an end-to-end model or is that the point where we need to design two parts: a generative model in raster space and one deep learning-based vectorizer?</p>
<p>We know from the DeepSVG paper by Carlier (2020) that it is technically possible to generate linear interpolations between two vector graphics with a different number of shapes. But are these interpolations as useful as interpolations in raster space?</p>
<div class="figure align-default" id="deepsvg-linear-interpolation">
<a class="reference internal image-reference" href="../../_images/deepsvg_linear_interpolation.png"><img alt="DeepSVG linear interpolation" src="../../_images/deepsvg_linear_interpolation.png" style="width: 700px;" /></a>
<p class="caption"><span class="caption-number">Fig. 6 </span><span class="caption-text">Linear interpolation between two vector graphics as made possible by DeepSVG; image taken from the <a class="reference external" href="https://arxiv.org/abs/2007.11301">DeepSVG paper</a></span><a class="headerlink" href="#deepsvg-linear-interpolation" title="Permalink to this image"></a></p>
</div>
</div>
<div class="section" id="current-vectorizers-to-convert-png-to-svg-perform-poorly">
<h3>5. Current vectorizers (to convert PNG to SVG) perform poorly<a class="headerlink" href="#current-vectorizers-to-convert-png-to-svg-perform-poorly" title="Permalink to this headline"></a></h3>
<p>Currently available vectorizing (“tracing”) tools perform extremely poorly. Purely the fact that an image has a low resolution can lead to situations where the pixels are traced like steps. Furthermore, there is no contextual awareness: identical objects of different sizes may get represented differently in the resulting SVG.</p>
<div class="section" id="examples">
<h4>Examples<a class="headerlink" href="#examples" title="Permalink to this headline"></a></h4>
<p><strong>Example 1:</strong></p>
<p>The image below shows the “Octocat”, Github’s mascot.</p>
<div class="figure align-default" id="github-mark">
<a class="reference internal image-reference" href="../../_images/GitHub-Mark.png"><img alt="github" src="../../_images/GitHub-Mark.png" style="width: 300px;" /></a>
<p class="caption"><span class="caption-number">Fig. 7 </span><span class="caption-text">Github’s mascot Octocat</span><a class="headerlink" href="#github-mark" title="Permalink to this image"></a></p>
</div>
<p>Note the ellipses on the octocat’s tentacle-tail in the figure above.
Vectorizing a low resolution version of that section of the image leads to a poor result, see the figure below:</p>
<div class="figure align-default" id="vectorizing-still-poor">
<a class="reference internal image-reference" href="../../_images/vectorizing_still_poor.png"><img alt="vectorizing_still_poor" src="../../_images/vectorizing_still_poor.png" style="width: 900px;" /></a>
<p class="caption"><span class="caption-number">Fig. 8 </span><span class="caption-text">Current vectorizers are still performing poorly. In this case, the used tool was <a class="reference external" href="https://vectormagic.com/">vectormagic</a> – still one of the better tools out there and not free.</span><a class="headerlink" href="#vectorizing-still-poor" title="Permalink to this image"></a></p>
</div>
<p>The low resolution results in the smaller ellipses to have corners in the vector output. It appears that the vectorizer has no understanding that the resolution of the input is low and pixel edges and corners should not be traced too closely. The vectorizer also has no contextual understanding that there is a symmetry in these shapes: all of these are ellipses of different sizes.</p>
<p><strong>Example 2:</strong></p>
<p>In another example, see figure below, an image loses its perfect symmetry through vectorization.</p>
<div class="figure align-default" id="vectorizing-destroy-symmetry">
<a class="reference internal image-reference" href="../../_images/vectorizing_destroy_symmetry.png"><img alt="vectorizing_destroy_symmetry" src="../../_images/vectorizing_destroy_symmetry.png" style="width: 400px;" /></a>
<p class="caption"><span class="caption-number">Fig. 9 </span><span class="caption-text">Example from <a class="reference external" href="https://github.com/jankovicsandras/imagetracerjs">imagetracerjs</a></span><a class="headerlink" href="#vectorizing-destroy-symmetry" title="Permalink to this image"></a></p>
</div>
<p><strong>Example 3:</strong></p>
<p>Vector graphis, such as SVG, can make use of linear and radial gradients. This can produce visually pleasing results, such as this logo below:</p>
<div class="figure align-default" id="edge">
<a class="reference internal image-reference" href="../../_images/edge.svg"><img alt="edge" src="../../_images/edge.svg" width="300px" /></a>
<p class="caption"><span class="caption-number">Fig. 10 </span><span class="caption-text">Microsoft Edge logo designed as a vector graphic in SVG format; <a class="reference external" href="https://www.npmjs.com/package/&#64;browser-logos/edge">source</a></span><a class="headerlink" href="#edge" title="Permalink to this image"></a></p>
</div>
<p>Since this logo was designed as an SVG, we know it is possible to design such a logo as a vector graphic.
Here is the SVG code for the logo above:</p>
<div class="figure align-default" id="edge-logo-svg">
<a class="reference internal image-reference" href="../../_images/edge_logo_svg.png"><img alt="edge_logo_svg" src="../../_images/edge_logo_svg.png" style="width: 900px;" /></a>
<p class="caption"><span class="caption-number">Fig. 11 </span><span class="caption-text">SVG code of the Microsoft Edge logo</span><a class="headerlink" href="#edge-logo-svg" title="Permalink to this image"></a></p>
</div>
<p>However, current vectorizers have a hard time converting a rasterized version of the logo back to a vector graphic.
These algorithms have no understanding of what (potentially overlapping) shapes are required and what shapes need to have what type of linear or radial gradients. There are too many parameters at play for pre-Deep Learning algorithms to work well.</p>
<div class="figure align-default" id="vectorized-edge-logo">
<a class="reference internal image-reference" href="../../_images/vectorized_edge_logo.png"><img alt="vectorized_edge_logo" src="../../_images/vectorized_edge_logo.png" style="width: 300px;" /></a>
<p class="caption"><span class="caption-number">Fig. 12 </span><span class="caption-text">Screenshot of the vectorized Microsoft Edge logo; vectorized using <a class="reference external" href="https://www.vectorizer.io">vectorizer.io</a>. The vector result is much more complicated than the original SVG and no where near the original in terms of quality.</span><a class="headerlink" href="#vectorized-edge-logo" title="Permalink to this image"></a></p>
</div>
</div>
<div class="section" id="summary-of-limitations-of-current-vectorizers">
<h4>Summary of limitations of current vectorizers<a class="headerlink" href="#summary-of-limitations-of-current-vectorizers" title="Permalink to this headline"></a></h4>
<ul class="simple">
<li><p>No context-aware tracing</p>
<ul>
<li><p>E.g. not prior classification of objects and if they should be round or edgy</p></li>
<li><p>E.g. the bottom of a heart shape becomes round (–&gt; incorrect points / edges)</p></li>
</ul>
</li>
<li><p>No resolution-aware tracing</p>
<ul>
<li><p>At low input resolution, tracers start to follow pixel steps</p></li>
<li><p>E.g. this shape was likely a circle even though it is now a 20x20 pixel blob</p></li>
</ul>
</li>
<li><p>Potentially incorrect trade-off between accuracy and number of points</p>
<ul>
<li><p>Vectorizers rarely produce elegant results with minimal number of points</p></li>
</ul>
</li>
<li><p>Over-use of points vs. a smart use of bezier curves and their handles</p></li>
<li><p>No or poor detection of colour gradients</p></li>
<li><p>No consideration of symmetry</p></li>
<li><p>Tracers do not understand how humans compose images from multiple shapes (potentially overlapping, potentially half-transparent), e.g. shadows</p></li>
<li><p>SVGs just have too many parameters and an intractable complexity for current vectorizers to correctly guess them all given a raster image</p></li>
</ul>
<p>Because current vectorizers are performing poorly (and will hopefully soon be replaced by smarter Deep Learning-based algorithms), there is a need to output vector graphics directly. Instead of attempting to output raster images and only then vectorize them.</p>
<p class="caption"><span class="caption-number">Fig. 14 </span><span class="caption-text">Linear interpolation between two vector graphics as made possible by DeepSVG; image taken from the <a class="reference external" href="https://arxiv.org/abs/2007.11301">DeepSVG paper</a></span><a class="headerlink" href="#deepsvg-linear-interpolation" title="Permalink to this image"></a></p>
</div>
</div>
</div>
Expand Down
2 changes: 1 addition & 1 deletion docs/03_related_work/further_academic_papers/Das_2020.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -507,7 +507,7 @@ <h1>Das et al. (2020) | BézierSketch: A generative model for scalable vector sk
</div>
<div class="figure align-default" id="das-2020-cover-png">
<a class="reference internal image-reference" href="../../_images/das_2020_cover.png"><img alt="Das 2020 paper" src="../../_images/das_2020_cover.png" style="width: 500px;" /></a>
<p class="caption"><span class="caption-number">Fig. 45 </span><span class="caption-text">Screenshot of the <a class="reference external" href="https://arxiv.org/pdf/2007.02190.pdf">Das et al. BézierSketch paper</a></span><a class="headerlink" href="#das-2020-cover-png" title="Permalink to this image"></a></p>
<p class="caption"><span class="caption-number">Fig. 47 </span><span class="caption-text">Screenshot of the <a class="reference external" href="https://arxiv.org/pdf/2007.02190.pdf">Das et al. BézierSketch paper</a></span><a class="headerlink" href="#das-2020-cover-png" title="Permalink to this image"></a></p>
</div>
</div>

Expand Down
2 changes: 1 addition & 1 deletion docs/03_related_work/further_academic_papers/Das_2021.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -507,7 +507,7 @@ <h1>Das et al. (2021) | Cloud2Curve: Generation and Vectorization of Parametric
</div>
<div class="figure align-default" id="cloud2curve-2021-png">
<a class="reference internal image-reference" href="../../_images/cloud2curve_2021.png"><img alt="Cloud2Curve paper" src="../../_images/cloud2curve_2021.png" style="width: 500px;" /></a>
<p class="caption"><span class="caption-number">Fig. 46 </span><span class="caption-text">Screenshot of the <a class="reference external" href="https://arxiv.org/pdf/2103.15536.pdf">Das et al. Cloud2Curve paper</a></span><a class="headerlink" href="#cloud2curve-2021-png" title="Permalink to this image"></a></p>
<p class="caption"><span class="caption-number">Fig. 48 </span><span class="caption-text">Screenshot of the <a class="reference external" href="https://arxiv.org/pdf/2103.15536.pdf">Das et al. Cloud2Curve paper</a></span><a class="headerlink" href="#cloud2curve-2021-png" title="Permalink to this image"></a></p>
</div>
</div>

Expand Down
Loading

0 comments on commit dcf540c

Please sign in to comment.