Sourced from: https://feedback.techsmith.com/techsmith/topics/1080p-recording
The webcam recording in Camtasia is factory-limited to 720p for performance reasons, but can be user-modified. There are two preferences needed to implement the change, maxCameraWidthRecordingScreen and maxCameraWidthNotRecordingScreen.
To set the preferences, implement the following in a Mac Terminal:
$ defaults write com.techsmith.camtasia2.plist maxCameraWidthRecordingScreen -int 1920$ defaults write com.techsmith.camtasia2.plist maxCameraWidthNotRecordingScreen -int 1920
This should be done when Camtasia is not running.
NOTE regarding the Camtasia Application:
If you are running the non App Store build, the plist is
~/Library/Preferences/com.techsmith.camtasia2.plist .
If you are running an App Store build, it is ~/Library/Containers/com.techsmith.camtasia2/Data/Library/Preferences/com.techsmith.camtasia2.plist .