Duplicate elements in a hash

I have a problem with hashes at the moment. I have classes that are immutable and only contain one element, when I add two different classes with the same data to the hashset, I get them both in the set. This is weird because I've overloaded Equals and GetHashCode for both the base class and the superclass.

public abstract class Contact :IEquatable<Contact>
{
    public readonly BigInteger Id;

    public Contact(BigInteger id) { this.Id = id; }

    public abstract bool Equals(Contact other);

    public abstract int GetHashCode();

    public abstract bool Equals(object obj);
}

      

        
        
        
      

    

And the inheritance class:

public class KeyOnlyContact :Contact, IEquatable<KeyOnlyContact>
{
    public KeyOnlyContact(BigInteger id) :base(id) { }

    public override bool Equals(object obj)
    {
        if (obj is KeyOnlyContact)
            return Equals(obj as KeyOnlyContact);
        else if (obj is Contact)
            return Equals(obj as Contact);
        else
            return (this as object).Equals(obj);
    }

    public override bool Equals(Contact other)
    {
        if (other is KeyOnlyContact)
            return Equals(other as KeyOnlyContact);
        else
            return (this as object).Equals(other as object);
    }

    public bool Equals(KeyOnlyContact other)
    {
        return other.Id.Equals(Id);
    }

    public override int GetHashCode()
    {
        return Id.GetHashCode();
    }

      

        
        
        
      

    

As you can see, all the real work is deferred to BigInteger, which is the identifier. This is a .net class and I have confirmed that I am not getting a duplicate if I just add BigInteger to the hashset.

To clarify:

BigInteger a;
HashSet<Contact> set;

set.add(new KeyOnlyContact(a));
set.add(new KeyOnlyContact(a));

set.Count == 2