This is probably a healthy concern.
I think there might be some good ways to hedge one's bets, though.
Use LLMs for rubber ducking, code scanning and review, rather than code generation.
Keep LLM code contributions minimal and unremarkable, too.
Don't make them load-bearing. If the code is central to the program, it's too unique.