James is talking about unfaithfulness to the Lord.
Since our Lord is SpiritSoulAndBody it is possible to be unfaithful -> to His Spirit -> to His Soul and -> to His Body
A person is unfaithful to the Holy SpiritOfChrist Jesus by entertaining evil spirits.
A person is unfaithful to the Soul of Christ by denying His Word.
A person is unfaithful to the Body of Christ by adding works to Grace.