This error is received because 1 byte equals 1 character only if you use the latin-1 character set. If you use utf8, each character will be considered 3 bytes when defining your key column. If you use utf8mb4, each character will be considered to be 4 bytes when defining your key column. Thus, you need to multiply your key field's character limit by, 1, 3, or 4 (depending on which character set is being used) to determine the number of bytes the key field is trying to allow. If you are using uft8mb4, you can only define a maximum of 191 characters for a native, InnoDB, primary key field. Just don't breach 767 bytes.

MySQL assumes worst case for the number of bytes per character in the string. For the MySQL 'utf8' encoding, that's 3 bytes per character since that encoding doesn't allow characters beyond U+FFFF. For the MySQL 'utf8mb4' encoding, it's 4 bytes per character, since that's what MySQL calls actual UTF-8.

Looking at an example:

Assuming that utf8mb4 is being used, this means that 4 bytes are needed for every character. Attempting to have a key column be 255 characters long will fail with this error because those 255 characters actually takes up 4 times that amount of bytes, i.e. 255 * 4 = 1020. However as mentioned, the maximum length supported by InnoDB is 767, therefore you would need to change this from:

ALTER TABLE `mytable` ADD UNIQUE ( column1(255));

to

ALTER TABLE `mytable` ADD UNIQUE ( column1(191));