Gaining control of a town of your culture won't come with any loyalty penalties but a penalty will come into effect as soon as your newest conquest belongs to a different cultural group. One important piece of the loyalty puzzle is culture. Town Loyalty in Bannerlord determines how likely (or unlikely) the population is to rebel, so it's pretty vital that you do everything you can to keep its drift in check.
