keyboard - macOS remap keys as modifiers

I use a keyboard layout called 3l (three layout) which is a layered keyboard layout. Here is the main layer (the behaviour when no modifier keys are pressed):

Now, when I am holding down the ' on a normal keyboard (sym in 3l), I am in "symbols" mode and the rest of the keyboard behaves differently. Here's a picture of this mode:

Effectively, this symbols key is like a special Shift which allows me to access special characters.

Additionally, I have a numbers layer which has the same effect as the symbols layer and is activated by the / key, but has cursor and number-pad controls instead:

---

The person I forked my layout from already got the layout working in Linux and Windows (using AutoHotKey). I am trying to port it to macOS as well. So far I have created a 3l.keylayout file (see it here) and have very basic functionality working.

What's Working

1. The keys in normal mode work fine.


2. If I press the symbol modifier key, the first character I press is outputted as the correct symbol. For example, if I press the symbol key and then O, / is outputted.


3. The same goes for the cur/num key for the numberpad output.

What's not Working

1. If I press and hold the symbol modifier key, and type multiple characters, only the first one actually outputs the symbol. (Same goes for the cur/num key as well.)


2. If I press and release one of the modifier keys and then press another key, the symbol/number is outputted. This should not be the case. When I release the modifier key, I want it to go back to normal mode (just like the Shift key).


3. The cursor manipulation keys do not work reliably in all applications and they also have the issue of not allowing repeated keys when they do work.

Is there any way for me to get the functionality I described above in macOS (Sierra and later)?

Note: I have also tried to remap ' and / to Right Option and Right Control using Karabiner Elements and then attempted to create a new keymap in 3l.keylayout that is activated by those keys. This did not work, however. It screwed everything up and I got into a perpetual control/option key situation.

Answer

I was able to fix this issue using Karabiner-Elements' complex modifications.

Here is my complete karabiner.json file.

Under profiles -> rules -> manipulators, I added two manipulators. One for sym and one for cur.

I copied the sym manipulator below:

{
    "conditions": [
            "name": "sym",
            "type": "variable_if",
            "value": 0
        },
        {
            "name": "cur",
            "type": "variable_if",
            "value": 0
        }
    ],
    "from": {
        "key_code": "quote",
        "modifiers": {
            "optional": [
                "any"
            ]
        }
    },
    "to": [
        {
            "set_variable": {
                "name": "sym",
                "value": 1
            }
        }
    ],
    "to_after_key_up": [
        {
            "set_variable": {
                "name": "sym",
                "value": 0
            }
        }
    ],
    "type": "basic"
}

Then I went through every key setting, i.e. when the key is pressed with no modifier, the sym modifier, and the cur modifier.

This works flawlessly.