What determines whether alleles are dominant or recessive?
The alleles have different DNA sequences. Because the sequence of DNA contains information to make products, different sequences can lead to different products. One advantage of diploid species is that there are two copies of every sequence. If one sequence makes a "faulty" or non-functional product, it would be called a loss-of-function allele. However, there is likely another sequence that produces a "correct" or functional product. For most genes a single wild-type (usually the normal / functional) allele is capable of producing enough product for the cell, resulting in a dominant phenotype. However, if both copies of the gene are loss-of-function alleles, there will not be any functional protein and the recessive phenotype will be observed.
An example from pea plants
Mendel studied pea plants in which the peas could be round or wrinkled, with wrinkled being the recessive characteristic. The gene that determines this trait (originally represented as R) has since been identified and named SBE1 (for starch-branching enzyme). The wild-type sequence encodes an enzyme that catalyzes a chemical reaction of carbohydrates to form branched chains of starch in plants. When SBE1 protein is present and functional, the peas produce branched starch and exhibit a round shape. When SBE1 protein is absent, the amount of branched starch is reduced, but the levels of disaccharides are higher, resulting in increased water absorption. Later, this water will be lost and the pea will become wrinkled.
Molecular studies of DNA in round (RR) and wrinkled (rr) plants revealed an insertion of about 800 base pairs in an exon of the SBE1 gene in the rr plants (Bhattacharyya et al, 1990). The insertion is derived from a transposon and disrupts the SBE1 protein-coding sequence. Therefore, the RR plants have two copies of a functional SBE1 allele (SBE1+/+) to make functional SBE1 enzyme, while rr plants have two copies of the SBE1 gene in which the sequence is disrupted (SBE1-/-) and cannot produce any functional SBE1 enzyme.
So what happens in a heterozygous Rr (SBE1+/-) plant? These plants have one chromosome with the SBE1+ allele and one chromosome with the SBE1- allele. The SBE1- allele is still transcribed, but does not code for the functional protein during translation. The SBE1+ allele is transcribed and translated into a functional enzyme, starch-branching occurs, and the peas have the round phenotype. Although the total amount of SBE1 enzyme is less than homozygous dominant peas, it is sufficient for the round phenotype and therefore dominant. In this case, molecular discoveries 125 years after Mendel's work reveal the reason for the dominance of the R allele over the r allele.
References
Bhattacharyya MK, Smith AM, Ellis TH, Hedley C, Martin C. The wrinkled-seed character of pea described by Mendel is caused by a transposon-like insertion in a gene encoding starch-branching enzyme. Cell. 1990 Jan 12;60(1):115-22. doi: 10.1016/0092-8674(90)90721-p. PMID: 2153053.