767 bytes is the stated prefix limitation for InnoDB tables - its 1,000 bytes long for MyISAM tables.

When using UTF-8 encoding, the maximum length of the columns in the key needs to factor in the additional characters used to support UTF-8.

MySQL assumes worst case for the number of bytes per character in the string. For the MySQL 'utf8' encoding, that's 3 bytes per character since that encoding doesn't allow characters beyond U+FFFF. For the MySQL 'utf8mb4' encoding, it's 4 bytes per character, since that's what MySQL calls actual UTF-8.

You can get the key to apply by specifying a subset of the column rather than the entire amount. Assuming that column 1 was 255 characters long, unfortunately this would fail because 255 characters actually takes up 4 times that amount of characters, i.e. 255 * 4 = 1020 characters. However as the maximum length supported by InnoDB is 767, you would need to change this from:

ALTER TABLE `mytable` ADD UNIQUE ( column1(255));

to

ALTER TABLE `mytable` ADD UNIQUE ( column1(191));