What you describe sounds very much as if the electrical contacts inside the phone's USB socket have gotten bent, or the connection to the phone's circuit board is damaged. If the latter, a skilled person with the correct tools (very fine tipped soldering tool and thin electronic solder) might be able to repair the board, but if the former, the only reliable repair would be to replace the circuit board on which the socket is mounted. With a phone a generation or two old like the Note 4, it's likely this can be done at reasonable cost with tested used parts if you can do the work yourself, but it's likely the labor cost will make it more cost effective to replace the entire phone if you have to pay someone to do the work.
To prevent a repeat of this sort of damage, be sure not to apply any force to the USB cable plug other than pushing it directly into the socket, or pulling it straight out. Any significant force bending the plug toward either flat surface will tend to either bend the contacts or shell, or break the connections to the circuit board.