In order to get the full picture of the event community and the full consent of people joining it online, everyone is required to sign up for the event.
During sign up ticket holders are also accepting the terms of service and the rights to message them in the context of the event is correctly obtained.
For these reasons it is not possible to add attendees and ticket holders manually to the event. More information about networking between attendees.
Issue free tickets with restricted access to bypass payment
If you need to issue tickets to someone for free, you can simply create an additional ticket type. Set the ticket to free and restrict the access with a ticket code. You can then forward the direct ticket link to the attendees, and they will need to properly sign up and accept the terms to register.