propagate_anchors should resolve component references in special layers (intermediate, alternate, etc.)

[anthrotype]

In both the old anchor_propagation.py and the new one ported from fontc (#1011), anchor propagation is only performed for 'master' layers. To get the anchors from a given component, we search the base glyph for a layer with the same layerId, however this only works for master layers. If we can't find a layer in the referenced base glyph that has the same layerId as the layer where a component is defined, we skip that component and move on.
Therefore, composite glyphs inside non-master, special layers such as intermediate (brace) or alternate (bracket) layers, or the various color layers, do not currently inherit any anchors from their component glyphs.
Ideally we should match the built-in Glyphs.app implementation and propagate the anchors for special layers as well.

In order to do that, we need to fully implement the logic for resolving component references, which is encapsulated in the GSComponent.componentLayer property from Glyphs.app Python scripting API.

As Georg explained in #853, a componentLayer is not the parent layer where a component is defined, but the layer which the component is pointing to in the base glyph, either one of the existing layers or a new one interpolated as needed.

The logic to resolve component references goes as follows (currently only point 1. is implemented):

1. First, search the component's base glyph for a layer with the same layerId; if we find one, this must be a 'master' layer for only the latter kind have the same layerId in all the glyphs - non-master layers have a unique layerId, not useful for matching across glyphs.

2. Else, look for a layer with the same associatedMasterId and a matching layerKey. Wait - what's a layer key now?! It's what the private GSLayer.layerKey() method returns in the Glyphs.app python interface: a string that concatenates a layer's optional attributes such as the intermediate coordinates, alternate layer's axis range, whether it's a color layer or "color palette" or bitmap strike or SVG color layer. It may look like {600} or Regular [550‹wght‹700] or Color 1 or a combination of these. Also note, the layerKey is always None for master layers (they don't need one) and for non-master/non-special (attribute-less) layers like the newly created ones used as draft or backup (they aren't compiled into the font anyway).
   So, when step 1 - direct match by layerId - is unsuccessful, step 2 is to search for a layer in the base glyph that has the same layerKey as the component's parent layer as well as the same associatedMasterId; the latter is equally important, as the layerKey is not sufficient by itself for a match. There's a private obj-c method named GSGlyph.layerForKey_masterId_ that one can call from the Glyphs.app's Python interface, which uses these two string parameters, layerKey and the master id associated with a layer, to find the matching layer in another glyph if any (or None if none is found); it's used internally by Glyphs.app to implement the componentLayer property, Georg said.

3. If the above steps failed and the component's parent layer is an intermediate layer, it means that the composite glyph where our component is defined has more (intermediate) layers than the base glyph it is pointing to - otherwise we would have found a match already (or it might also mean the base glyph has an intermediate layer with the same coordinates but it's associated with a different master...). No problem. To resolve this component reference, we need to create a new layer by interpolating the base glyph at the intermediate location of the component's parent layer. Easy peasy? I wish.

4. Else, if step 1 and 2 didn't find a match and step 3 doesn't apply to us (the component's parent layer is not an intermediate that we can interpolate), then fall back to the layer in the base glyph which has the same associatedMasterId as the component's parent layer. This is guaranteed to be there (unless the master ids are bogus) because all glyphs have >= len(font.masters) layers, and all layers (must) have a valid associatedMasterId.

So, the missing pieces for fully resolving component references, are:

• implement the layerKey method, to peform step 2. This is relatively straightforward and can be tested by comparing with the output of Glyphs.app Macro panel.

• write code to build variation models for any glyph to interpolate new layers with it, in step 3. At some point we did have some of that code: @simoncozens made Build intermediate layers with non-intermediate components #971, and later removed in remove resolve_intermediate_components from preflight, no longer needed #992 because this was no longer needed. However even that is not complete because it doesn't take into account the fact that a given Glyphs.app's glyph may define multiple variation models: the presence of alternate ('bracket') layers means that there can be multiple versions of the same glyph (to be activated within some axis ranges) that are incompatible with one another, each with it's own set of source layers. Alternates can even contain additional intermediate layers, so it's not necessarily one model (same locations, same scalars) applied to distinct groups of source layers, but it could be different models altogether, defined over more or less source locations. Depending on which axis range the desired interpolation coordinates fall within, one needs to use one of these alternate models.

Once we have all of that, we should be able to take any component and find or create the layer it is referencing, even when the component is in a non-master, special layer; and we can complete the propagate_anchors code to properly match Glyphs.app.

Now for layerKey, I already have a branch, which I'll PR soon.
But for the glyph layer interpolator the work is more involved than I can commit to at this stage. I do have some WIP code but it's not in a reviewable state. I don't know if I'll have time to finish it myself right now, so I opened this issue and may also push it to a WIP branch for later.

[schriftgestalt]

I can add the layerKey method (in the Glyphs3 branch).

[anthrotype]

I can add the layerKey method

@schriftgestalt thanks Georg, but since I have already written one, do you mind taking a look at #1018?

[schriftgestalt]

Here is my version: 94cbdff
it is slightly more complicated, specially with multiple color palette layers.

[anthrotype]

thanks Georg, that's helpful, I hadn't noticed the dedup handling of multiple color layers with same palette index.

I noticed that your code falls back to the layer's self._name when it's orphan (no parent font) or when it has no special attributes; this doesn't match what I see returned from within Glyphs.app Macro panel when I call the built-in layerKey() method. This returns None in both these cases. Even if I set a name on the orphan or attribute-less layer before calling layerKey(), it always returns None.

[anthrotype]

another thing I noticed is that Glyphs.app's built-in layerKey() method always returns None for master layers. In your branch, you are returning the master name + optional bracket axis rules. I think we should stick to whatever Glyphs.app actually does, the main use for this layerKey() is for resolving component references

[anthrotype]

To resolve this component reference, we need to create a new layer by interpolating the base glyph at the intermediate location...

turns out this is actually more complicated than I had initially thought. I can't even make it work properly from within Glyphs.app.
The problem is when a glyph has both intermediate layers and alternate layers and you want to interpolate a new layer at some axis coordinates. My understanding was that you first collect all the glyph's source layers whose min/max axis ranges all span the given coordinates, excluding those that don't apply; then use these to build a variation model and do the interpolation. But even in Glyphs.app sometimes this works, other times it leads to failed interpolations (export errors about incompatible outlines or components disappearing altogether).

I made a test font where I combine intermediate and alternate layers and reference these from composite glyphs to see how they work together here:

SpecialLayers.glyphs.zip

@schriftgestalt maybe we can schedule a meet to talk about this? I'm interested in this not just for a more accurate anchor propagation, but also because at some point in fontc we'll need to implement support for alternate layers (currently only intermediates are supported) and this is relevant for that as well.

[schriftgestalt]

another thing I noticed is that Glyphs.app's built-in layerKey() method always returns None for master layers.

There is one more check I missed in my implementation. The code in my branch is from a method called: _layerKey() and that is called like this:

- (NSString *)layerKey {
	if (!_layerKey && _attributes.count > 0) {
		_layerKey = [self _layerKey];
	}
	return _layerKey;
}

So all layers that don’t have any attributes will return nil.

[schriftgestalt]

I had a look at your file.
There are some interesting combinations. I need to go through them in more detail.

I found on important thing already: I think right now, the {600} brace layer in "I" should show the "G" shape and should not look at the bracket layers. When building the final font, all layer are put into two different glyphs, all the ones without bracket layers go into the default glyph (it is actually more complicated, see below (1)) and all the bracket layers go into extra glyphs. So the {600} goes into the default glyph as it doesn't have any brackets.

One thing that I’m not fully using glyph.layerGroups() to compute the interpolation. Here is a small script that shows the layer groups (you also can check them with "Show master Compatibility" in the View menu.)

glyph = Layer.parent
groups = glyph.layerGroups()
for group in groups:
	for layerId in group:
		layer = glyph.layers[layerId]
		print(layer)
	print("____")

(1) I use the following pattern a lot:

That means the two masters are not compatible. the light shows the default shape (e.g. the dollar with the continuous stroke and the Bold only has the two bits at the top and bottom.). That way you see the popper shapes in font view (otherwise you would see the completely filled in dollar).

[anthrotype]

thanks for the tip about glyph.layerGroups(), I did notice that the "Show master Compatibility" view was in fact getting the groups of compatible layers correctly. However sometimes the interpolation is failing (e.g. in the glyph "P" which I marked as non-export). We can take a look together when you're back from holidays, thanks!